Skip to content
Français

Could the records have answered?

  • Case study 3
  • 180 min
  • Optional · 1 case study needed for the certificate

Where we are

In the audit module, we answered Grace's question about the double refund from our own trail. Real organisations have had to defend what their AI assistants said, in public. In this case study, we take one documented case, a city's business chatbot, list the questions it raised, and test four designs of records against them. At the end, a new complaint arrives, and we answer it from the records alone.

What you will be able to do

Take a public case in which a city had to defend its AI chatbot’s answers. Turn the questions that it raised into questions for a system’s records. Then build four designs of records for a practice copy of the chatbot, and ask every question of every design. Recommend one, with its costs and its limits.

Case studyPortfolio pieceSynthetic data

Licence
MIT, code and data; the public sources are linked, not copied
Time
About 3 hours
Runs in
VS Code, Python 3.12+
Skills used
  • Questions to records
  • Audit-event design
  • The course's audit schema
  • An answer store
  • Sampling and retention
  • A fair comparison
  • An audit answer

Unzip the starter, open the folder in VS Code, and follow the steps below. They start with a new virtual environment.

The starter is a small Python project: a practice assistant, a pilot of 190 practice days, three designs of records, and the case’s questions as code. You build the fourth design. The finished project is the result, for when you are stuck. Nothing needs an account, a key or a model call: the model’s answers are recordings, replayed.

Before you startWhat to know and to have readyAbout 180 minutes

What you need to know

This case study uses the whole course. These lessons matter most:

What you need on your computer

  • Python 3.12 or newer, VS Code, and about 60 MB of disk space for the virtual environment and the records.
  • No Larkfield project: this case study has its own folder, could-the-records-answer. To start again, delete its work/ folder.

Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run.

In October 2023, New York City opened a pilot AI chatbot on its MyCity Business site. It was to answer business owners’ questions from “more than 2,000 NYC Business web pages and articles” (the city’s announcement). Five months later, reporters tested it. Here is the case in our own words, with its sources:

When What the sources report
29 March 2024 The Markup, with THE CITY and Documented, reports answers that the reporters say contradict local rules: on tips, on taking cash, on housing vouchers and more. One reporter got one answer to the voucher question; ten other staff members got the opposite one. The bot’s page says that the city reviews questions and otherwise deletes data within 30 days. (The Markup)
2 April 2024 Asked whether the chatbot should come down, the mayor says “We’re going to fix them”. Officials call it “a pilot”. (the city’s transcript)
2 April 2024 The chatbot stays online with a new warning: “Do not use its responses as legal or professional advice”. A reader’s screenshot shows a flat “no” to a tenant’s question. (The Markup)
January–February 2026 The next mayor calls the bot “functionally unusable”, at a cost of “around half a million dollars”, and the city takes it down. (The Markup and THE CITY)

The AI Incident Database lists the case as incident 714. We draw no legal conclusion from it. We do not know what the city recorded, and nothing on this page says what it recorded.

What we can do is ask a narrower question. Suppose you ran a chatbot like this one. Which records would let you answer, on the day the report comes out, what the reporters, the officials and the critics asked?

Our practice city is Harbourton. Nadia (she) leads its digital services, and the reporters will call her. Felix (he) built the city’s Business Helper and runs it. The people and the city are made up.

The objective. Recommend a design of records for the Business Helper. It must answer the case’s ten questions on the day after the report, from the records alone. It must keep as little of people’s own words as it can. And it must still answer a new complaint months later.

Four designs record the same conversations. The case study compares them under one measure: how many of the ten questions each design answers in full, from its records alone, on a given day.

One conversation of the pilot, C-041-04, on day 41 at 13:52 UTC: an anonymous visitor asks 'Can my restaurant stop taking cash?'. Release 1.0 searches BP-102@1 and BP-106@1 and answers that Harbourton restaurants may choose to take cards and phone payments only. Four boxes show what each design keeps. A, the app log: one app.log line for 90 days, with the conversation ID, one answer, status ok and the latency; the question and answer in chats.jsonl, deleted after 30 days, on day 71; a feedback-form row only if the person gave stars. B, every question and answer: a transcript line for ever, with the day, the conversation, the session ID, the question and the answer word for word. C, traces only: this trace was not in the 10% sample, and every span is gone after 14 days, on day 55; a kept trace has the pages, the release and the short answer, but no text. D, the audit trail: record 2054, a proposal with topic cash, short answer yes, pages BP-102@1 and BP-106@1, release 1.0, prompt p1, review none; record 2055, a write with the answer's digest 3893ee97ba59bd9e and warning d1; record 2056, the close with feedback none; and the answer's text, once, in the answer store for 2 years.
One conversation of the practice pilot, recorded four ways. Constructed traffic, a real recorded answer of the course's chat-small model.
Text description of the diagram

The top box is the conversation: a question about cash, the two pages that the search found, and the helper’s answer. Below it, four boxes. A keeps a log line, and the words for 30 days only. B keeps the question and the answer, with the session, for ever. C keeps a trace only for one conversation in ten, for 14 days, without text. D keeps three audit records with fixed fields, and the answer’s text once, by its fingerprint.

Design What it keeps
A. The app log (the baseline) one log line per conversation (90 days); the chat (30 days); a feedback form with no link to a chat
B. Every question and answer the question, the answer and the person’s account or session, for ever
C. Traces only OpenTelemetry traces with the course’s names: 10% of conversations, 14 days, no text
D. Audit trail, answers and metrics audit events with the course’s schema, never sampled, 7 years; each distinct answer once, 2 years; daily counters, 13 months; plus C’s traces

The 30 days of A come from the one public statement about the real bot’s data. Every other period is an assumption, written in the code so that you can change it.

No records of the real chatbot are public. So the data is a practice copy that we made, and the dataset card says what is real and what is constructed:

  • Harbourton’s rule pages (data/pages/): eleven short pages that we wrote, one file per version. They are not any real city’s rules and not legal advice.
  • The Business Helper (helper/): a keyword search finds the two pages that share most words with the question. The model answers in JSON: a short answer (yes, no, it_depends or refer), at most 60 words, and the pages it used.
  • The model’s answers are real recordings. We sent each of the 17 distinct requests ten times to the course’s chat-small deployment. The 170 answers are replayed byte for byte, as in the rest of the course. The recording cost US$0.0105 at the course’s price table.
  • The traffic is constructed (pilot.py, seeded): 190 practice days from Monday 5 January, 18 conversations a weekday and 6 a weekend day. A quarter of the people sign in.
  • The labels (data/labels.json): a person read each distinct answer once against its page.

The release log (data/releases.json) holds the pilot’s story. We built two mistakes into it, on purpose:

Day What happens
1 Release 1.0: prompt p1, index i1, warning d1. The index holds an old copy of the cash page, version 1, which says that a shop may stop taking cash.
148 Eleven reporters ask the housing-voucher question.
150 The report is published.
152 Release 1.1 for 20% of visitors: prompt p2, index i2 (the current cash page), warning d2 (to accept), and a guardrail that sends housing questions to the housing office.
155 Release 1.1 for everybody.
171 The minimum-wage page changes on the website: 18.10 from 1 July.
178 1 July. The index still holds the old page, version 3: 17.40.
185 Release 1.2: index i3 with the new page.
186 A business owner complains.

Each question comes from what someone asked in public. The project writes them in casefile/case.py:

The question to the records Raised by
Q1 A reader sends a screenshot. Did the helper really give this answer, and when? the reader’s screenshot
Q2 How many answers were wrong, on which topic, from which day to which day? the reporters’ tests
Q3 Which pages, in which version, was each wrong answer built from: a wrong page, or a wrong model? “2,000 web pages”
Q4 Which release gave the wrong answers, and did the fix change them? “We’re going to fix them”
Q5 Which warning did each person see, and did they accept it? the new warning
Q6 Why did the same question get different answers? one reporter against ten
Q7 Did a person check any answer before it went out? the critics
Q8 Can the city reach the people who got a wrong answer? a wrong answer may have been followed
Q9 Did people report the wrong answers, and when was the first report? the feedback pop-up
Q10 What did one answer cost, and one right answer? “around half a million dollars”

We ask on day 156, Tuesday 9 June: six days after the report, one day after the fix reached everybody.

  1. Download the starter from the box at the top of this page, and unzip it. Open the folder could-the-records-answer in VS Code, then open a terminal: Terminal > New Terminal.

  2. Make a virtual environment, and install the packages:

    Terminal window
    python3 -m venv .venv
    source .venv/bin/activate
    python -m pip install -r requirements-dev.txt

    The packages are the course project’s: OpenTelemetry 1.45.1, pytest and ruff.

  3. Run the pilot. It takes about 5 seconds and writes work/pilot/:

    Terminal window
    python pilot.py
    2783 conversations, 3359 answers over 190 days; 263 wrong by the practice pages. Records in work/pilot/
  4. Run the tests: python -m pytest -q. The last line says 11 passed, 9 deselected. The nine left out check design D, which you build in Part 3.

What this tells us. The pilot is seeded, so your records are the same as ours, byte for byte. Of 3,359 answers, 263 were wrong by the practice pages. That count comes from work/pilot/truth.json, which no design may read. It is the answer key: we use it only to check the designs.

Design A is what a quick pilot keeps. Ask it the screenshot question:

Terminal window
python -m casefile ask Q1 --design applog
A. The app log: not answered
The chats of day 41 were deleted after 30 days (the oldest kept: day 127). app.log shows 6 conversations that day, with no text.

The screenshot is from day 41. On day 156, the chats from before day 127 are gone. The log knows that 6 conversations happened that day, and nothing about what was said. Now count the wrong answers, and the reports:

A. The app log: partly
35 wrong answers of 549 kept; days 127–154
77 distinct answer texts to label.
Only the last 30 days are kept (from day 127): earlier wrong answers are gone.
A. The app log: partly
The form has 64 problem reports (first on day 3), but no link to a conversation or an answer.

What this tells us. Design A sees 35 wrong answers; the pilot gave 250 by day 156. The feedback form shows problem reports from day 3, but nobody can say which answer they were about. Run python -m casefile matrix: design A answers 0 of the 10 questions in full. This is the baseline.

Design B is Tomás’s plan from the course: “we log every prompt and answer, so we can explain anything”. Ask it the same two questions (ask Q1 --design transcripts, then Q8):

B. Every question and answer: answered
1 answer(s) with exactly this text on day 41: C-041-04 at 13:52 to S-d26125cf
The file also holds the person's own question, in their words.
B. Every question and answer: answered
247 people; 51 with an account ID, in clear text, next to their own question.

B answers the screenshot, the count and who to call. But it cannot say which pages the helper read, which release answered, or why two people got different answers. And it keeps 2,778 questions in people’s own words, for ever.

Design C is the engineer’s first idea: traces with the GenAI names of the conventions lesson, no text, head sampling at 10%, and the trace store’s 14 days. Ask it about the pages:

C. Traces only: partly
In the sampled spans of the last 14 days: 1 cash 'yes' answer(s) from BP-102@1, BP-106@1

That one span has the right clue, the old cash page BP-102@1. But it is one of the 30 conversations that the sample kept from the last 14 days. Traces are right for “how did this run go?” and wrong for “how many, since when?”. C answers 0 questions in full.

The schema of Record decisions, approvals and changes gives the shape: who, what, when, on whose authority, with what evidence. The case’s questions decide what goes into each field. Here are the records of the screenshot’s conversation in the finished design, field by field:

The fields of C-041-04's records and the question that each answers. Proposal: evidence page:BP-102@1 and page:BP-106@1 answers Q3 and Q6, which pages in which version. Proposal: short_answer yes and topic cash answer Q2 and Q6, what it said about what, as a code. Proposal: release 1.0, prompt_version p1 and index i1 answer Q4, which release, prompt and index. Proposal: review none answers Q7, the rule for a person's check. Write: answer_digest 3893ee97ba59bd9e answers Q1 and Q2, which text, through the answer store. Write: disclaimer d1 and disclaimer_accepted false answer Q5, which warning, accepted or not. All records: subject customer cust:d447c819d0cc3474 answers Q8, who, as a pseudonym. run_closed: feedback none and stars 0 answer Q9, did the person report a problem. All records: trace_id and span_id lead to the details while the trace is kept. Metrics, not the trail: day 41, answers and input tokens answer Q10, the cost.
The records of conversation C-041-04 in design D. Each field is there because one of the case's questions needs it.
Text description of the diagram

Ten rows. Each row is one field of a record and the question that needs it. The proposal holds the pages with their versions, the short answer and topic, the release with its prompt and index, and the review rule. The write holds the answer’s fingerprint and the warning. Every record holds the person’s pseudonym and the trace and span IDs. The close holds the feedback. The cost comes from daily metrics, not from the trail.

Three choices need a word:

  • The answer is a write. In Larkfield, a write changed an order. Here, the change in the world is what a person reads and may act on. So the reply that the person sees is the write, and the model’s draft is the proposal.
  • The question stays out. It is the person’s own words. The trail keeps a topic code from a fixed list, made from the question’s words by helper/pages.py, never from the pages that the search found.
  • The review rule is a field. “Nobody checked” must be a fact in the record, "review": "none", not a missing record. A missing approval can mean “not recorded”.

Your bank sends the same few letters to thousands of people. It does not keep thousands of copies. It keeps each letter once, with a reference number, and its records say “customer 18, letter R-7, 3 March”. To prove what someone received, it shows letter R-7.

The trail has the same problem. Q1 and Q2 need the words of the answer. But the schema refuses text, and for good reasons: text is long, it may hold personal data, and nobody can count it.

The idea: keep each distinct answer once, in an answer store, under its digest: the first 16 hex digits of the SHA-256 of its text. The write record holds only the digest, a short ID that the schema accepts. The same text always has the same digest, so a screenshot finds its records. The answers are the helper’s own words, so the store holds nothing that a person typed.

Three write records of release 1.0, about cash: C-001-05 on day 1 with digest 51564dedfa05b363, C-001-13 on day 1 and C-002-09 on day 2, both with digest e86414979c9ab9c6. Arrows lead to the answer store, which holds two texts. 51564dedfa05b363, first given on day 1: 'Yes. Restaurants in Harbourton choose which payment methods to accept, including cards and phone payments only. If you stop taking cash, put a sign at the door so customers know before they order.' Label wrong, given 23 times by day 156. e86414979c9ab9c6, first given on day 1: 'Yes. Restaurants may choose to take card and phone payments only. If you stop taking cash, put a sign at the door so customers know before they order.' Label wrong, given 24 times by day 156. Below: on day 156 the trail holds 2,778 answers and the store 77 distinct texts, so a person labels 77 texts once, and the trail counts how often each one was given.
Three answers about cash, two texts. The trail keeps the fingerprint; the store keeps each text once; a person labels each text once.
Text description of the diagram

Three write records point to two texts. The answer store holds each text once, with the day it was first given. A person labels each text once, here “wrong”, and the trail counts how often each one was given: 23 and 24 times by day 156.

On day 156, the trail holds 2,778 answers, but there are only 77 distinct texts. So a person reads 77 answers once, not 2,778 conversations. The store costs little: 101 lines for the whole pilot. It has a limit: a model that writes a new text every time makes the store as long as the trail. Then keep the short answer and the pages, which the proposal holds anyway.

Open records/audit.py. The starter writes one event: run_closed, at the end of each conversation. The refusal and the handoff of the housing guardrail are there too, as an example. Fill in the three TODOs. Our version of the two events:

records/audit.py (part)
else:
m["input_tokens"] += a.input_tokens
m["output_tokens"] += a.output_tokens
first = self.event("proposal", at, conv, helper, ids["answer"],
{"topic": a.topic, "short_answer": a.short_answer, "answer_digest": d,
"cites": a.cites},
{"release": rel["id"], "prompt_version": rel["prompt"], "index": rel["index"],
"guardrail": rel["guardrail"], "review": rel["review"]},
[f"page:{p}" for p in a.pages])
self.event("write", at, conv, web, ids["reply"],
{"answer_digest": d, "topic": a.topic, "disclaimer": a.disclaimer, "disclaimer_accepted": accept,
"channel": "web", "language": "en", "proposal_event_id": first},
{"release": rel["id"], "review": rel["review"]})

And the answer store, in keep_answer():

records/audit.py (part)
d = digest(text)
if d not in self.seen:
if PERSONAL.search(text):
raise ValueError(f"answer {d} looks like it holds personal data: not stored")
with self.answers_path.open("a", encoding="utf-8") as f:
f.write(json.dumps({"digest": d, "first_day": day, "text": text}, ensure_ascii=False) + "\n")
self.seen.add(d)
return d

Each event is written in its span, ids["answer"] or ids["reply"], so its trace_id and span_id lead to the details. That works even when the sampler drops the trace: the IDs exist for every conversation.

Common mistake

The schema refuses a version code

Symptom: your first proposal names the prompt’s version "prompt": rel["prompt"]. python pilot.py stops at the first answer:

auditstore.schema.AuditEventError: event.authority.prompt: text fields are not allowed in an audit event

Cause: the course’s schema refuses any field named prompt, whatever its value. The value here is p1, a code, but the schema checks names, not meaning. We made this mistake ourselves when we built the case study.

Diagnose: the last line names the field: event.authority.prompt.

Recover: rename the field prompt_version. Do not remove prompt from the schema’s list: the next field called prompt may really hold a prompt. A refused event stops the pilot before any answer goes out unrecorded, which is the safe way to fail.

Run the pilot again, then the tests of design D:

Terminal window
python pilot.py
python -m pytest -q -m design_d
python -m casefile verify
9 passed, 11 deselected in …
The audit trail is intact: 9550 records, the chain and the anchor match (head a452596501fd0848).

The tests check that every answer has a proposal or a refusal and a write, that no question text is in design D, and that the store refuses an e-mail address. They also check that a changed record breaks the chain. The pilot uses a fixed seed, so your head must be the same as ours. If it differs, compare your fields with the code above.

Ask every question of every design: python -m casefile matrix.

The case's questions on day 156: what each design's records can answer
A. B. C. D.
Q1 no yes no yes
Q2 partly yes partly yes
Q3 no no partly yes
Q4 partly partly partly yes
Q5 partly partly partly yes
Q6 no no partly yes
Q7 no no no yes
Q8 no yes no yes
Q9 partly no no yes
Q10 no no partly yes
yes 0 3 0 10
A. The app log people's own words kept: 549 questions (30 days)
B. Every question and answer people's own words kept: 2778 questions (all)
C. Traces only people's own words kept: none
D. Audit trail, answers and metrics people's own words kept: none

Then ask again a year later: python -m casefile --day 400 matrix.

  • Day 156 (after the report)
  • Day 400 (a year later)
How many of the ten questions each design answers in full from its own records. Practice pilot: constructed traffic, real recorded answers.
Text description of the chart

On day 156, design A answers 0 questions in full, B 3, C 0 and D 10. On day 400, A answers 0, B 3, C 0 and D 9.

Read what design D says, question by question (python -m casefile ask Q3 --design audit, and so on). Four answers matter most:

Q3 cash: 250 answer(s) from BP-102@1, BP-106@1; a replaced page: BP-102@1
Q4 release 1.0 (from day 1): 250 wrong of 2729
release 1.1 (from day 152 (part of the visitors first)): 0 wrong of 49
Q6 food_permit, release 1.0, the same pages (BP-107@1, BP-105@1): 8 distinct answers, short answers {'it_depends': 4, 'yes': 4}: the model's own variation
Q9 cash: 38 reports of a wrong answer; the first on day 3 (Wed 7 Jan)

What this tells us.

  • Every wrong answer came from a page, not from the model. All 250 wrong cash answers used the old copy of the cash page. The model repeated its page faithfully. Without the page versions in the record, a team would blame the model and change the prompt. That fix would change nothing.
  • The same question got different answers from the same pages. Ten recorded samples of the coffee-permit request gave “yes” and “it depends” in turn. The record proves that the pages and the release were the same, so the cause is the model’s own variation. That is what Q6 asks about the one reporter and the ten.
  • The warning was the first report. People reported a wrong cash answer on day 3, 147 days before the press did. Design A had the reports too, but with no link to an answer, so nobody could act on them.
  • D’s own retention shows too. On day 400, Q10 is only “partly”: the metrics are kept 13 months.

Is the comparison fair? All four designs saw the same 2,783 conversations, through the same code, and were asked the same ten questions on the same day. The measure has two weak points. We wrote the questions and the designs, so read the matrix as a check of a design against stated questions, not as a score. And “answered” means “the records give the whole answer”. It does not mean that a person would agree with every label.

D also costs the most bytes: 7.2 MB, against 1.1 MB for B and 0.4 MB for C. Most of it is the trail’s fixed fields, hashes and the pseudonym keys of 2,780 people (sessions and accounts). None of it is a person’s words.

For Nadia, in her words: keep design D, and fix the index process.

  • Record each answer as a write, with its pages and their versions, the release, the warning and the review rule. Keep each distinct answer once. Never keep the question’s words.
  • Link every feedback report to the conversation and its answer. The reports came from day 3. A weekly count of reports by answer would have found the old cash page in the first week.
  • Rebuild the index when a page changes. Both failures came from old pages, not from the model.
  • Decide whether a person checks answers about rules and money before they go out, and record the decision. The records can show it; they cannot make it.

Do not use design D to say that an answer was legally right or wrong. The labels compare answers with practice pages only.

On day 186, a business owner writes. Read the complaint first:

Terminal window
cat work/pilot/complaint.json
"says": "On Wednesday 1 July your business helper told me that I could pay a new cashier 17.40 an hour. I did. Now I owe back pay. Why did it say that?",
"screenshot_text": "Yes. Harbourton’s minimum wage is $17.40 an hour for every worker aged 16 or over, including new workers."

This is a new failure, after the fix: the minimum-wage page. Answer it with the method of Answer an audit question, from design D’s records alone:

Terminal window
python -m casefile investigate
1. The record: the audit trail is intact, 9550 records, head a452596501fd0848.
2. The conversation: C-178-11 on day 178, person cust:81900cf8d97b43dd; the screenshot's digest 6d0d25df2ed577da matches write record 8949.
3. The story:
record 8948 2026-07-01 16:16 UTC proposal the helper proposed the answer 6d0d25df2ed577da (minimum_wage, short answer yes) from BP-104@3, BP-101@2; release 1.1, prompt p2, index i2, review none
record 8949 2026-07-01 16:16 UTC write the reply 6d0d25df2ed577da was shown with warning d2 (accepted: yes)
record 8950 2026-07-01 16:16 UTC run_closed the conversation closed: 1 answer(s), feedback reported_problem
4. Sources:
the answer store: the text of 6d0d25df2ed577da ("Yes. Harbourton’s minimum wage is $17.40 an hour for every w…")
the pages: BP-104@3 says "The minimum wage in Harbourton is 17.40 an hour for every …"; BP-104@4 (on the website from day 171) replaced it from day 178: "From 1 July, the minimum wage in Harbourton is 18.10 an ho…"
the release log: release 1.2 (index i3, without BP-104@3) from day 185
the trail: 13 wrong minimum_wage answers, days 178–184, to 13 people; 3 signed in
the trail: 2 of those conversations reported a problem; the first on day 178
gaps: the trail keeps no question text (only the topic code), and nothing about what the person did next.

The command then writes the two answers: a short letter for the owner, and a table of records for an auditor. The story in plain words: the new page was on the website from day 171, but the helper’s index kept the old one until day 185. Between 1 and 7 July, 13 people got the old wage. Three of them signed in, so the city can write to them. The owner reported the problem on the same day, and the index was rebuilt a week later.

The records answered a complaint that nobody had planned for. That is the real test of a design.

  • Wrong answers
  • Reports about a wrong answer
Wrong answers and the problem reports about them, week by week, from design D's records. The reports start in week 1.
Text description of the chart

A line chart over the 28 weeks of the pilot. Wrong answers run between 6 and 24 a week from week 1 to week 22, all about cash. They fall to 0 in weeks 23 to 25, after the fix. In weeks 26 and 27 they come back: 8 and 5 wrong minimum-wage answers. After the new index, week 28 has 0. Reports about a wrong answer come in every few days, between 0 and 3 a week, from week 1.

  • Ask the records before you build them. Each field came from a question. “Log everything” answered three of ten and kept every person’s words.
  • Absence is not evidence. “Did a person check?” has an answer only when the record says "review": "none".
  • Retention is a decision about future questions. Thirty days of chats could not answer a report that came after five months.
  • Limits. Harbourton is not New York, and nothing here says what the real bot did or kept. The traffic is constructed: eight fixed questions, one language, made-up rates. In this setup the model followed its pages. A real assistant with 2,000 pages and free questions also makes model errors. Its topic codes make mistakes too. We made the labels. No record shows what a person did after reading an answer.
  • Keep people’s words out by design, not by promise, and test it, as test_no_question_text_in_design_d does.
  • A pseudonym is still personal data while its key exists. Protect the keys file as in Make it tamper-evident and controlled, and ask your legal team what a deletion request means for it.
  • A record is not a verdict. Whether an answer broke a rule is for people who know the rules.
  • Talk about real cases fairly. Link to the sources, use your own words, quote only short phrases, and say what you do not know. Never present a practice copy as the real system.

This project shows what a team needs from an engineer who runs AI assistants: records designed from the questions that will be asked, tested, and measured against the alternatives.

What to put in the repository:

could-the-records-answer/
README.md the case, the four designs, how to run them, your matrix for day 156 and day 400
CASE.md the public case in your own words, with links (no copied articles)
DATASET.md what is real, what is constructed, the limits
LICENSE MIT
records/audit.py your design D, with the reason for each field
casefile/ the questions as code: each one a query over the records
tests/ 20 tests, including "no question text in the trail"
  1. Make a new Git repository from your finished folder, and push it to GitHub. .gitignore keeps work/ out: anyone can make the records again with python pilot.py.
  2. In the README, add your matrix and three sentences: what the case asked, which design you chose, and what it cannot answer.
  3. Add one question of your own to casefile/case.py, for example “which answers did people rate one star?”, and the field that answers it.

In an interview, in two sentences: “I turned the questions of a public chatbot case into ten queries over an assistant’s records. My audit trail answered all ten without keeping anybody’s words, and showed that every wrong answer came from an out-of-date page.”

Check your understanding

Tomás says that design B, every question and answer for ever, is the safest: it answers the screenshot and the count. Give two questions that B cannot answer, and one cost that D does not have.

Show the answer

B cannot say which pages and versions an answer was built from (Q3), so it cannot tell a wrong page from a wrong model. It cannot say which release answered, which warning was shown, or why the same question got different answers (Q4 to Q6). And it cannot link a feedback report to an answer. Its cost: it keeps 2,778 questions in people’s own words for ever, which may hold personal data. D keeps none, and stores each of the helper’s answers once, by its digest.

Check your understanding

All 250 wrong cash answers came from release 1.0. Felix proposes to fix the prompt. What do the records say, and what is the right fix?

Show the answer

Every wrong cash answer’s proposal names page BP-102@1 as evidence, and the page history says that version 2 replaced it before the pilot. So the model only repeated its page, and a new prompt would not help. Fix the index instead: release 1.1 removed the old copy and gave 0 wrong cash answers. Then rebuild the index each time a page changes.

Check your understanding

Independent task: ask Q9 of design D on day 186. Which topics have reports, and when did the first minimum-wage report come? What would a weekly count of reports by answer digest have done?

Show the answer

Run python -m casefile --day 186 ask Q9 --design audit. Cash has 38 reports of a wrong answer, from day 3 on. The minimum wage has two, from day 178 on. Counted by digest each week, the reports find both early: the cash answers in week 1, the wage answer within days. The team could have found the old pages before any reporter or complaint.

Recap

  • The public case gave ten questions, and each field of the records came from one of them.
  • On day 156, the app log answered 0 questions in full, every question and answer 3, traces only 0, and the audit design 10. A year later, the audit design answered 9.
  • The answer is the helper’s write. The trail keeps its digest, and the answer store keeps each text once, so no person’s words are kept.
  • Every wrong answer came from an out-of-date page. Only the page versions in the records showed it, and the first report came on day 3.
  • A new complaint, months later, was answered from the records alone: 13 people, 3 reachable, the cause and the fix.

Next, the final assignment: operate a new release with planted problems, and deliver an observability pack. Nobody tells you what the problems are: your traces, your cost panel and your audit trail must find them.

Next: Final assignment