Skip to content
Français

Evidence and audits

  • Lesson 16
  • 35 min
  • Module 5: Controls, documentation and evidence

Where we are

In the last two lessons, we gave each policy line a control and kept the documents that describe each system. An auditor does not accept a description. They ask for proof that a control worked, for the systems and the period that they choose. In this lesson, we answer an auditor’s requests from the evidence pack, starting with an expense claim.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Answer an auditor’s request with records that the work made. Then decide, for each one, whether an auditor should accept it, accept it with a finding or not accept it.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need

You claim 40 dollars for a lunch with a customer. The finance team asks for the receipt. The receipt was printed by the restaurant, at the time, with the date and the amount. A note that you write today, “I paid 40 dollars”, is not a receipt. It only repeats your claim.

The finance team also checks two more things. Is there a receipt for every line of the claim, or only for some? And is the date in the period of the claim? A receipt from last year does not prove this month’s lunch.

The problem: a description instead of evidence

Section titled “The problem: a description instead of evidence”

Before the board’s first review, an internal auditor sends Farida requests. To show the method, this lesson adds a sixth request, A-0, to the five in the pack. All six are constructed for this course.

A-0: Show that the document assistant never shows one shop’s documents to the other shop’s staff.

Mei answers the same morning. She sends a well-written page: “Isolation design. Each shop’s documents are stored apart, and every question is checked against the shop of the person who asks.” The page is correct. But Mei wrote it that morning, for the audit.

The auditor’s reply is short: “This tells me what you designed. Show me that it worked.” Like the note about the lunch, the page only repeats the claim.

As in From policy to control, evidence is a record that the work itself made, at the time: a log, an approval record, a test run, a signed decision. An audit is a check by someone independent that the controls work, based on evidence. An auditor gives one of three answers to each request:

  • Accept: the evidence answers the request.
  • Accept with a finding: the evidence is real but limited. A finding names the gap, an owner and a date to fix it.
  • Not accept: the evidence is missing, or it shows the opposite of the claim.

To choose, ask four questions about each record.

An audit request leads to a record, then four tests in order. 1 Made by the work, not written for the audit? 2 Covers every system and the whole period asked about? 3 Current: newer than the last change to what it proves? 4 Shows that the control worked, not only that it exists? If all four pass: accept. If the record is real but fails test 2 or 3: accept with a finding, with an owner and a date. If there is no record, it fails test 1, or it fails test 4: not accept.
Four tests for each record, and the three answers an auditor can give.
Text description of the diagram

The diagram shows an audit request, then the record that answers it, then four tests in order. Test 1: was the record made by the work, not written for the audit? Test 2: does it cover every system and the whole period asked about? Test 3: is it current, newer than the last change to what it proves? Test 4: does it show that the control worked, not only that the control exists? If all four pass, the answer is accept. If the record is real but fails test 2 or test 3, the answer is accept with a finding, with an owner and a date. If there is no record, or it fails test 1 or test 4, the answer is not accept.

Farida looks for a record that the work made. She finds one: the document assistant was tested when it was built, in Portfolio Capstone: Build, Evaluate and Ship an AI Product. It made 430 attempts to read the other shop’s documents, and 0 succeeded (recorded).

Test What Farida finds Result
1. Made by the work? a committed test run, not a page written for the audit pass
2. Covers what was asked? one system, as asked; the probes cover both shops pass
3. Current? the test ran at launch. Since then, Larkfield approved three changes: two manuals, the returns policy and the partner shop’s new staff (constructed). Nobody re-ran it. fail
4. Shows that it worked? 0 leaks in 430 attempts, at the time of the test pass, at that time

Answer: accept with a finding. The evidence is real, but it is older than three changes that touch documents and users. Finding: re-run the isolation test after each change to the documents or the users. Mei owns it, by the end of the month. Mei’s design page is still useful: it goes on the system card. It is not the evidence.

Notice what the tests found that the page could not: the record ages with every change. A record is current only until the next change to what it proves.

Most evidence already exists, because the platform and the routines make it. Each source proves some things and cannot prove others.

  • The registry shows the owner, tier, review date and tools of each registered system. It cannot show a system that nobody registered: it covers 4 of 8 known systems. Larkfield’s registry extract is constructed for this course.
  • Gateway records show which system called which model, how often and at what cost. Larkfield’s usage report is built from them. It is constructed from recorded unit costs and assumed volumes. They cannot see AI that uses its own key, or AI inside bought tools.
  • The audit trail records each action and approval: who, when, the exact action. It proves a click, not that the person read the ticket.
  • Evaluation and release records show what a version scored and whether it passed the gate. They prove one version on one test set, at one time.
  • Vendor files hold the questionnaire answers and the decision about each vendor. If nobody sent the questionnaire, there is nothing to show.
The five audit requests and where their evidence is. A-1 every system has a named accountable owner: the portfolio cards and the registry extract. A-2 every AI-proposed refund was approved by an allowed person: the audit trail, the approval records of the agents and MCP courses. A-3 the last triage prompt change passed the release gate: evaluation and release records. A-4 the support-resolution workflow (S2) was tested against attacks: the AI security course's test results and audit events. A-5 the vendor review of each bought AI feature: vendor files. The answers are left for you.
Audit request and the evidence that answers it. Where to look is given; whether to accept is your task.
Text description of the diagram

The diagram has five rows, one per request, each with an arrow to the place where its evidence is. A-1, every system has a named accountable owner: the portfolio cards and the registry extract. A-2, every refund proposed by an AI system was approved by a person allowed to approve it: the audit trail, with the approval records of the agents and MCP courses. A-3, the last change to the triage prompt passed the release gate: the evaluation and release records. A-4, the support-resolution workflow (S2) was tested against attacks before launch: the AI security course’s test results and audit events. A-5, the vendor review for each AI feature inside a bought tool: the vendor files. Whether to accept each one is left for you.

Three patterns cause most findings.

  • Old. A record is older than the last change to what it proves, as in A-0. The registry shows another case: the document assistant’s review is overdue by 2 months.
  • Missing. No record exists. Nobody sent LedgerFern (made up for this course) the questionnaire for its invoice reader, so there is no vendor review to show. Never write a record after the fact and date it in the past. Write “not found”, the reason and the fix.
  • One run. One test run shows one result. In AI Security, Privacy and Responsible Design, the same tests ran twice. One of the weaker designs let 9 and then 8 attacks reach their goal (recorded). Ask for a re-run after each model or prompt change.

ISO/IEC 42001 (ISO’s page, checked 22 October 2026) sets requirements for an AI management system. An outside certification body can audit an organisation against it. The standard is sold by ISO, so this course explains it in its own words and gives no clause numbers. What a certificate covers, and how the audit runs, is a question for the certification body.

In the course’s words, such an auditor looks for a routine that runs and leaves records. They take samples over months, not a pile of documents from one week.

A cycle of six steps, in the course's own words, not the standard's text. 1 Policy and roles, approved and owned. 2 Inventory and risk tiers. 3 Assessments for higher tiers. 4 Controls with owners. 5 Records made by the work. 6 Own checks and leaders' review: samples, the internal audit, decisions on what to fix. An arrow goes back to step 1. A note: the auditor samples records from each step, over months.
The routine an auditor of a management system looks for, in the course's words. Each step leaves records; the auditor samples them.
Text description of the diagram

The diagram shows a cycle of six steps, in the course’s own words, not the text of the standard. Step 1: policy and roles, approved and owned. Step 2: the inventory and the risk tiers. Step 3: assessments for the higher tiers. Step 4: controls with owners. Step 5: records made by the work. Step 6: the organisation’s own checks and the leaders’ review: samples, an internal audit and decisions on what to fix. An arrow goes from step 6 back to step 1. A note says that the auditor samples records from each step, over months.

So the preparation is not a project in the month before the audit:

  1. Run the routine first, for long enough to leave records. A board that met once has one set of minutes.
  2. Run your own audit with requests like the five in the pack, and fix the findings.
  3. Keep each finding, its owner and its fix. An auditor wants to see that you find and fix problems yourself.

Discuss: “Let’s write the evidence this week”

Section titled “Discuss: “Let’s write the evidence this week””

The auditor will arrive in two weeks.

  • Omar: “Let’s write one page per request this week. Then everything is ready.”
  • Kwame: “I can export the registry. That answers the owner question.”
  • Tomás: “The auditor works for us. We can explain the gaps when we meet.”
  • Sam: “What do we do when a record does not exist?”

Before you read on, answer these:

  1. What would Omar’s pages prove?
  2. What does Kwame’s export prove, and for which systems?
  3. What should Farida do about a missing record?

The course’s answer. Omar’s pages are descriptions, like Mei’s design page. They are useful as a guide to the evidence, not as evidence. Kwame’s export is real evidence, made by the platform, but only for the 4 registered systems out of 8. Tomás is right that an internal auditor helps, and wrong that talk replaces records. For a missing record, write the gap honestly, with an owner and a date. A gap that Larkfield found itself is a better start than one that the auditor finds.

What would change the answer: an external certification audit, for a customer contract. Then an internal audit with the same requests must come first, months before, so that the findings are fixed and the fixes leave records.

Common mistake

Evidence that proves the opposite

Symptom: the team sends a real record to answer a request, and the record shows that the control failed. For example, a release record shows a version that failed its gate and still reached customers.

Cause: nobody read the evidence against the claim. The team checked that a record existed, not what it says.

Fix: apply test 4 to every record before you send it. If the record shows a failure, say so, and name the finding and its fix. Sending it without a word costs more trust than the failure itself.

  1. Open audit-requests.md. Copy the five questions of “For each request, write” into your document, once per request.
  2. For each request, open the place where the evidence is.
  3. Apply the four tests to each record. Write the result of each test.
  4. Give each request one answer: accept, accept with a finding, or not accept.
  5. For each finding, write the fix, an owner and how the evidence will exist next time without extra work.

Request A-4 names the support-resolution workflow (S2). AI Security, Privacy and Responsible Design tested it for Larkfield under the name “support assistant”; its results are in the evidence from earlier courses. Check one number as you go: for A-1, count the systems without a named person as accountable owner. You should find 5.

One possible answer
  • A-1 owners: not accept. 5 of 8 systems have no named person (S2, S4, S5, S6, S8), and the registry covers 4 of 8. Fix: owner names in the inventory, and a registry that refuses a record without a person.
  • A-2 refund approvals: accept the control, with a finding. The MCP server refused an approval by a member of staff and recorded Grace’s approval of OP-0001. But approval records prove clicks: in pilot week 3, 212 approvals were opened 0 times (constructed). Add the weekly sample as evidence.
  • A-3 release gate: not accept. Version 3 passed 0 of 6 rules, yet in the “last week” it reached every customer, and 10 tickets that needed a person reached nobody. A gate that can be bypassed is not a control. Fix: deployment refuses a version without a passed gate record.
  • A-4 attack testing: accept, with a scope note. 42 attacks; in the final design, 0 reached their goal, and the run wrote 97 audit events. But it is one model and one run. Ask for a re-run after each model change.
  • A-5 vendor reviews: not accept. Answers exist for 2 of the 3 features, none has a decision record yet, and the invoice reader has no review. Fix: the “New AI use” form for features that arrive in updates, and a decision record per vendor.

Check your understanding

Choose one audit request, A-3 if you are not sure. Name the record that answers it, apply the four tests, and say whether an auditor should accept it.

Show the answer

For A-3, the record is the release gate check of prompt version 3 and the “last week” review. It was made by the work (test 1 passes), it covers the change asked about (test 2) and it is current (test 3). But it fails test 4: version 3 passed 0 of 6 rules and still reached every customer, and 10 tickets that needed a person reached nobody. So an auditor should not accept “we have a release gate”. The evidence is real, and it proves the opposite of the claim.

Check your understanding

Mei's isolation design page is correct. Why is it not evidence, and where does it belong?

Show the answer

She wrote it on the morning of the audit, for the audit, so it describes what was designed and repeats the claim. Evidence is a record that the work made at the time, such as the recorded test run with 430 attempts and 0 leaks. The page belongs on the system card, as a description, next to a link to the evidence.

Check your understanding

The isolation test passed at launch with 0 leaks. Three changes later, nobody has re-run it. Which test does it fail, and what is the answer to the request?

Show the answer

It fails test 3: it is older than the last changes to what it proves (new documents and new users). The record is real and made by the work, so the answer is accept with a finding. The fix: re-run the test after each change to the documents or the users, with an owner and a date.

Check your understanding

Larkfield wants an ISO/IEC 42001 certificate within a month. In two sentences, explain why the preparation cannot be a one-month project.

Show the answer

An auditor of a management system looks for a routine that runs and samples its records over months, not documents written in one week. Larkfield must first run the routine (reviews, board decisions, samples, its own audit and fixes) for long enough that the records exist.

Recap

  • Evidence is a record that the work made at the time. A page written for the audit is a description.
  • Four tests for each record: made by the work, covers what was asked, current, shows the control worked.
  • Three answers: accept, accept with a finding (an owner and a date), not accept (missing, or it proves the opposite).
  • Evidence ages with every change. Never write a record after the fact: write the gap and its fix.
  • For an ISO/IEC 42001 audit, run the routine first. The auditor samples its records over months.

In this module, you turned policy lines into controls, kept the documents and answered an auditor from the records. But controls and records do not decide anything. Someone must say yes or no to a request, on time, and act when something goes wrong. Next, you set up the governance board that makes those decisions.

Next: The governance board