Review a published AI register
- Case study 3
- 120 min
- Optional · 1 case study needed for the certificate
Where we are
In this course, we built Larkfield's inventory and tiers from its own system cards and registry. The registry said that the triage service was low, and the course's method said high. The Government of Canada publishes a register of this kind for everyone: 40 algorithmic impact assessments, each with an impact level. In this case study, we read that register as an inventory, tier ten of its systems with the course's method, and find what a published record leaves out.
Case study 3 of 3 · AI Governance: Policies, Risk and AccountabilityReview a published AI register
1. What governance is for
2. Laws and standards in plain words
3. Inventory, risk tiers and impact assessment
4. Policies people can follow
5. Controls, documentation and evidence
What you will be able to do
Review a real, published register of automated decision systems. You build an inventory from public records, tier ten systems with the course’s method and compare your tiers with the published levels. Then you say what a published record leaves out, and what Larkfield should copy.
The case folder has a README and a source card. records.csv has one row per published record, and record-answers.csv has five answers that the course read from each one. The records themselves stay on the government’s portal.
Before you startWhat to know and to have ready
What you need to know
This case study uses the whole course. These lessons matter most:
- The fields of an inventory row, from The AI inventory.
- The course’s risk-tier method and its two questions before scoring, from Risk tiers and proportionate controls.
- What an impact assessment is for, from Impact assessments.
- Who must publish these records, from Canada, the US, the UK and others.
- Evidence against a description, from Evidence and audits.
The page teaches one new idea: a self-assessed level.
What you need
- A browser and a spreadsheet or a document tool.
- The case folder: its README, the source card, records.csv and record-answers.csv.
- Your copies of the inventory template and the risk-tier table.
The situation
Section titled “The situation”Farida, Larkfield’s risk and compliance lead, has an inventory of eight systems and a tier for each one. Nadia, the executive sponsor, sends her a link with one question:
“The Government of Canada publishes a register of its automated decision systems. What can we learn from it for our own inventory? And what does a published register leave out?”
Larkfield, Nadia’s question and the team are constructed for this course. The records are not: they are real public records, and every count on this page comes from them.
The register exists because of a rule. The federal Directive on Automated Decision-Making applies to federal institutions, not to private companies (Treasury Board of Canada Secretariat, official source, checked 22 October 2026). Before an institution uses an automated decision system, it must complete and publish an Algorithmic Impact Assessment (AIA). The AIA gives the system an impact level from I to IV. So Larkfield has no duty here. It reads the register to learn from it.
What you will make
Section titled “What you will make”You will make a two-page review: a ten-row inventory built from published records, a table “published level against our tier”, three findings and one recommendation for Larkfield.
Text description of the diagram
Eight steps, in order. First, the records: 40 published, with their licence and the source card. Second, the first checks: 38 records have results, and they use 5 versions of the questionnaire. Third, explore the levels, the scores and the answers. Fourth, the baseline: take each published level at its word. Fifth, our reading of ten records with the course’s method and its two questions. Sixth, one measure: which records get a close look? Seventh, a recommendation for Larkfield’s inventory. Eighth, the method on new input: the newest record, then Larkfield’s triage service.
The records
Section titled “The records”The portal’s collection “Algorithmic Impact Assessment” had 40 records when the course read it. Each record is a page on the Open Government Portal with a results document. The document has the department’s answers to a questionnaire, with points for each answer. It also has the impact level and the requirements of that level.
First checks
Section titled “First checks”Before you count anything, check what the files cover:
- 40 records, 38 with results. Of the other 2, one is the record of the questionnaire tool itself, and one has no PDF results document. Count 38, not 40.
- 5 versions of the questionnaire. The column
tool_versionshows them. 26 records use version 0.10.0. The questions changed between versions, so the scores can mean different things. - One number, two labels. Most results documents call the raw score “Raw Impact Score”. 3 newer ones call it “Raw Score”. The course’s first script read only the first label, so it left 3 cells empty. An empty cell means “not found”, not 0, so the course opened those documents, found the numbers, and changed the script to read both labels. Now all 38 rows have a raw score.
- One answer disagrees with its own points. The newest record answers “No” to “Will the system use personal information as input data?”, with 4 points. The other records give 4 points for “Yes” and 0 for “No”.
The course downloaded each results document again and compared its SHA-256 fingerprint with the pdf_sha256 column. All 38 matched, so the file describes the documents that are online.
What this tells us. The register is real and current, but it is not one clean table. It is 38 documents from 11 organisations, published from 2018 to 2026 with 5 versions of a form.
Explore what the review needs
Section titled “Explore what the review needs”The levels
Section titled “The levels”Text description of the chart
A column chart with four columns. Level I: 15 records. Level II: 23 records. Level III: 0. Level IV: 0.
What this tells us. Every published system is at level I or II, the two lowest of four. Yet the records include immigration decisions, benefits for veterans, passports and a hiring tool. Immigration, Refugees and Citizenship Canada publishes 13 of the 38. A register shows who publishes. It does not show who uses automated decisions and does not publish.
The scores behind the levels
Section titled “The scores behind the levels”Each level comes from a current score. The records give it, but not where one level ends and the next begins.
Text description of the chart
Four rows of dots. Level I, version 0.10.0: scores from 25 to 31. Level II, version 0.10.0: from 34 to 63. Level I, other versions: 24. Level II, other versions: 27, 33, 34, 42, 43, 45, 47, 48, 49, 49, 56.
What this tells us. Within one version, the two levels do not overlap. Across versions, they do: a score of 27 is level II in an older record and level I in a version 0.10.0 record. So compare levels, not scores, and never compare scores across versions. In 10 records, the current score is lower than the raw score.
What the department answered
Section titled “What the department answered”Three answers in record-answers.csv matter here:
- Automation. “Full automation” in 2 records, “partial automation” in 25. 11 older records were not asked. Both fully automated systems are at level I.
- Peer review. Each record prints the requirements of its own level. All 15 records at level I say “Peer review: None”. All 23 at level II list experts to consult.
- A responsible executive. Only 3 records, all in version 1.0.1, have a field for the assistant deputy minister responsible for the program.
The new idea: a self-assessed level
Section titled “The new idea: a self-assessed level”Think of buying a house. The seller fills in a form about it: any water damage, any old wiring? The form is useful, but it is the seller’s form. A buyer still pays an inspector, who looks at the house and writes what was found.
A self-assessed level is a level that comes from the owner’s own answers. The answers have points. The points give a score, the score gives a level, and the level decides the requirements. Nobody outside scores the system before the level exists. The level is useful: it is fast, the same for everyone and public. But it measures the answers, not the system. Larkfield’s registry tier is the same kind of thing: the triage team gave its own system “low”.
Text description of the diagram
The department answers the questionnaire, and each answer has points. The impact answers give the raw impact score. The mitigation answers, such as consultation, data checks and recourse, join it to give the current score. In 10 of 38 records, the current score is lower than the raw score. The current score gives the impact level, from I to IV. The record then prints the requirements of its level: no peer review at level I, and peer review at level II. A dotted line leads to what the record has no field for: measured results, complaints, a review date, and whether the system is still in use.
So read a published level as the seller’s form: a good start, and a claim to check. As Evidence and audits said, a description is not evidence.
A baseline: take the level at its word
Section titled “A baseline: take the level at its word”The simplest review copies the published level into the course’s tiers. Level I is low, level II is medium, and levels III and IV are high.
On all 38 records, this baseline gives no high tier at all, because no record is above level II. If Farida worked this way, the hiring tool and the passport system would get the light checks of a meeting-notes tool.
What this tells us. The baseline is cheap. But a review that only copies the level adds nothing: the level is already published.
The alternatives
Section titled “The alternatives”Two other ways to review a record:
- The course’s method. Score the four factors from the record’s own answers: harm, people affected, autonomy and data. Then apply the two rules. Score what the system does alone, not what a person checks later, as for Larkfield’s triage service.
- The method and the two questions. First ask the method’s two questions. Is the system’s area one that a law calls high-risk? The EU AI Act lists eight such areas in its Annex III (Regulation (EU) 2024/1689, official source, checked 22 October 2026). Examples are employment, migration and border control, and essential public services. And does it decide about a person with no person involved? A yes sends the record to the specialist first.
The EU law does not apply to the Government of Canada. Here, the first question only sorts records. It is not a legal finding about any of them.
The course chose ten records: five at level I and five at level II, easy cases and hard ones. Their factor scores are the course’s reading of each record’s answers, not the department’s view.
One measure: which records get a close look?
Section titled “One measure: which records get a close look?”The level and the tier use different scales, so “do they agree?” is a weak question. A better question decides the work: which records get a close look before use? A close look means a high tier, or the specialist first. The measure was chosen before the scoring.
| Record (short name) | Published level | Our factors (harm, people, autonomy, data) | Our tier | The two questions | Close look: baseline, method, method and questions |
|---|---|---|---|---|---|
| Staffing: automated interview scores | I | 3 + 3 + 2 + 2 = 10 | high | employment | no, yes, yes |
| Old Age Security: absences to investigate | I | 2 + 3 + 1 + 2 = 8 | medium | public benefits | no, no, yes |
| Police: transcription of recordings | I | 2 + 2 + 1 + 3 = 8 | medium | law enforcement | no, no, yes |
| Social Insurance Number: automated issue | I | 2 + 3 + 3 + 3 = 11 | high | decides alone | no, yes, yes |
| Border: export compliance dashboard | I | 1 + 2 + 1 + 2 = 6 | low | no | no, no, no |
| Access-to-information request website | II | 1 + 2 + 2 + 1 = 6 (one score unclear) | low | no | no, no, no |
| Immigration: spouse eligibility approvals | II | 2 + 3 + 3 + 2 = 10 | high | migration and border control | no, yes, yes |
| Veterans: mental health benefit | II | 3 + 3 + 3 + 3 = 12 | high | public benefits; decides alone | no, yes, yes |
| Veterans: claim summary tool | II | 2 + 3 + 1 + 3 = 9 | high | public benefits | no, yes, yes |
| Passports: facial recognition | II | 3 + 3 + 2 + 3 = 11 | high | biometrics | no, yes, yes |
| Close look before use | Baseline (the level as is) | The course’s method | The method and the two questions |
|---|---|---|---|
| Of the ten records | 0 | 6 | 8 |
What this tells us. The baseline sends none of the ten to a close look. The method sends 6, and the two questions add 2 more. Our tier is higher than the baseline for 8 records, the same for 1 and lower for 1. So the method does not raise everything. The access-to-information website is level II, and our reading is low: 1 + 2 + 2 + 1 = 6.
Two records explain most of the gap. The Social Insurance Number system answers “full automation” and is level I. It decides alone about a number that people need for work: 2 + 3 + 3 + 3 = 11, high. The spouse eligibility model approves eligibility alone, and officers make the final decision later. It never refuses. The department scored it level II; we score what it does alone: 2 + 3 + 3 + 2 = 10, high.
Discuss: a hiring tool at level I
Section titled “Discuss: a hiring tool at level I”Farida shows the table at the team meeting. The staffing record is at level I: it scores candidates’ recorded interviews automatically.
- Tomás: “The government assessed it and published it. If they say level I, it is level I.”
- Mei: “Its own record checks ‘Employment’ in the list of decision areas. That adds one point.”
- Omar: “And in this pilot, an HR advisor reviews every candidate. So a person decides.”
- Sam: “Then why does our method say high?”
Before you read on, answer these:
- Score the four factors from the record. Which rule or question decides the tier?
- What does level I require, according to the record itself?
- What would you need to see before you accept level I?
The course’s answer. By our reading, harm is 3 and people affected is 3: the result touches access to a job. Autonomy is 2: it scores every candidate, and a person reviews. Data is 2: recorded interviews. The sum is 3 + 3 + 2 + 2 = 10, high. Before scoring, the first question already says yes: employment is an Annex III area. So in Larkfield’s process, this tool goes to the specialist first.
The record prints the requirements of level I: “Peer review: None”. Omar is right that a person reviews every candidate in the pilot, but that is an answer in a form. To accept level I, you need evidence. That means test results against human raters and across groups of candidates, and a date for the next review. The record describes these tests as planned, with no numbers.
What would change the answer: suppose the scores only scheduled interviews and decided nothing about who stays in the pool. Then harm would drop to 2 and autonomy to 1: 2 + 3 + 1 + 2 = 8, medium. The first question would still send it to the specialist.
The recommendation
Section titled “The recommendation”Copy the register’s good habits. One record per system. The record exists and is published, inside Larkfield, before the system is used. It keeps the answers that gave the tier, not only the tier. A questionnaire is quick and the same for every team.
Add what it leaves out. Every row needs four more things:
- A named accountable owner. Only 3 of 38 records have a field for the responsible executive.
- A second scorer for any self-assessed tier. The risk lead scores with the team, as in Risk tiers and proportionate controls.
- Results, not plans: the last evaluation, with its date.
- What happened after launch: complaints, corrections and the next review date.
A tier without evidence is an opinion, even when it is published.
End to end on new input
Section titled “End to end on new input”The newest record
Section titled “The newest record”The newest record in the file, published 14 August 2026, is the traveller compliance indicator of the Canada Border Services Agency. It is at level II, and you have not used it yet. Run the whole review on it:
- Checks. It uses version 0.10.0 of the questionnaire, with a raw score. It is the record whose personal-information answer disagrees with its points. Write that down as a question for the department, not as a fact.
- Inventory row. Purpose: indicators that an officer uses to release a traveller or refer them to a second examination. The record answers that its clients are “particularly vulnerable”. No accountable owner field, no review date.
- The two questions. Border control is an Annex III area: specialist first.
- The method. 2 + 3 + 1 + 2 = 8: medium. Autonomy is 1, because an officer decides.
- Close look. The baseline says no, the method alone says no, and the two questions say yes.
What this tells us. On new input, the review still finds what the level alone hides, and a gap in the record itself.
Larkfield
Section titled “Larkfield”Now turn the review on Larkfield. Its registry is a register too, with tiers that each team gave itself. The registry says that the ticket triage service is low. The method says 3 + 2 + 3 + 2 = 10, high. It is the only registered system where the two disagree. That is the same finding as the hiring tool: a self-assessed level, with no second scorer and no evidence attached.
Insights and limits
Section titled “Insights and limits”Insights.
- No published record is above level II, although the systems touch immigration, benefits, passports and hiring.
- A published level is a score of the department’s own answers. It measures the form, not the system.
- One measure, chosen first, made the comparison fair: a close look for 0, 6 or 8 of ten records.
- Larkfield’s registry has the same weakness, and the same fix: a second scorer and evidence.
Limits.
- The factor scores are our reading of public answers. A department knows its system better; a different reader may score one point differently.
- Ten records of 38 are a sample chosen by the course, not a random one.
- The register shows only what was published. It cannot show systems without a record, or say why none is above level II.
- The portal changes. Your copy of the file is the course’s extract; the live collection may have more records.
Responsible use
Section titled “Responsible use”- Review records, not people. The records name the public servants who completed them. Never copy those names; the licence does not cover personal information.
- Do not present our tier as the government’s. Say “the course’s reading of the published answers”, every time.
- No legal conclusions. The EU AI Act does not apply to these systems. The Annex III question only sorts them for a specialist. Whether a system meets the Directive is for the institution and its reviewers.
- Give the acknowledgement from the source card wherever you use the records.
Your task: write the review
Section titled “Your task: write the review”Write your own answer before you open the one below.
- Inventory. Fill ten rows of the inventory template from published records: the course’s ten, or ten of your choice. Mark each field as filled from the record, partly filled or missing.
- Tiers. Ask the two questions, then score the four factors with one reason each, from the record’s own answers. Mark any score that is unclear.
- Compare under one measure. For each record: close look or not, by the baseline, the method, and the method with the two questions.
- Three findings about the register as a whole, each with a count from the files.
- One recommendation for Larkfield’s inventory, and what would change it.
Show one possible answer
This is one defensible answer, not the only one. A different score is fine when you give the record’s answer that supports it.
To: Nadia. From: Farida. Subject: what Canada’s published AI register teaches our inventory.
The register. 38 of 40 records have results (Open Government Licence – Canada). Level I 15, level II 23, level III 0, level IV 0. 11 organisations; 13 records from one department.
Inventory: what a record fills. Name, purpose, data and autonomy: filled from the answers. Kind (built or bought), people affected, vendor, status (only the project phase) and tier (a level on another scale): partly. Accountable owner (except 3 records), technical owner and review dates: missing.
Ten records, published level and our tier:
| Record | Level | Our factors | Our tier | Close look? |
|---|---|---|---|---|
| Staffing: automated interview scores | I | 3 / 3 / 2 / 2 | high | yes |
| Old Age Security: absences to investigate | I | 2 / 3 / 1 / 2 | medium | yes |
| Police: transcription of recordings | I | 2 / 2 / 1 / 3 | medium | yes |
| Social Insurance Number: automated issue | I | 2 / 3 / 3 / 3 | high | yes |
| Border: export compliance dashboard | I | 1 / 2 / 1 / 2 | low | no |
| Access-to-information request website | II | 1 / 2 / 2 / 1 | low | no |
| Immigration: spouse eligibility approvals | II | 2 / 3 / 3 / 2 | high | yes |
| Veterans: mental health benefit | II | 3 / 3 / 3 / 3 | high | yes |
| Veterans: claim summary tool | II | 2 / 3 / 1 / 3 | high | yes |
| Passports: facial recognition | II | 3 / 3 / 2 / 3 | high | yes |
Findings.
- No record is above level II, and the 2 fully automated systems are at level I. The baseline gives a close look to 0 of 10; our method and the two questions give it to 8.
- The level is self-assessed. At level I, the record’s own requirements say “Peer review: None” (15 of 15). No record has a field for measured results, complaints or a review date.
- The register is uneven. 5 questionnaire versions from 2018 to 2026, scores that cannot be compared across versions, two labels for the same score and one answer that disagrees with its points.
Recommendation. Keep the register’s habit: one record per system, written before use, with the answers that gave the tier. Add four fields to every row: a named accountable owner, a second scorer, the last evaluation result with its date, and complaints and corrections since launch. Apply it first to the triage service, the one system whose registered tier the method contradicts.
What would change it. If the records began to report results and reviews, our second scorer could use them, and reviews would be faster.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This case study shows an employer that you can review a real register with a written method, and say what it cannot show. Risk, compliance, audit and public-sector teams look for this.
What to publish:
- Your ten-row inventory, with each field marked filled, partly filled or missing.
- The table “published level against our tier”, with the close-look measure.
- Three findings with their counts, and the recommendation.
- The acknowledgement: “Contains information licensed under the Open Government Licence – Canada.” Say that the tiers are your reading with a course method.
How to describe it in an interview, in two sentences: “I reviewed Canada’s 38 published algorithmic impact assessments as an inventory, and tiered ten of them with a written method. None is above level II, so I showed which systems a self-assessed level would let through without a close look, and what each record should add.”
Go further:
- Run your review on the UK’s algorithmic transparency records, published under the Open Government Licence. What do they have that the Canadian records do not?
- Open the live collection. Are there records newer than 14 August 2026? Review one.
Knowledge check
Section titled “Knowledge check”Check your understanding
Every published record is at level I or II. Give two reasons why that does not prove that the government's automated decision systems are low-impact.
Show the answer
First, the level is self-assessed: it is a score of the department’s own answers, and the record has no field for measured results, complaints or reviews. Second, the register shows only published records. It cannot show systems without a record. A third reason: the questionnaire changed over 5 versions, so the scores are not on one scale.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The spouse eligibility record says that officers make the final decision on every application. Why does the course still score its autonomy 3?
Show the answer
Score what the system does alone, not what a person checks later. The model approves eligibility by itself; the officer then screens for admissibility and decides. A wrong approval at the eligibility step is rarely overturned: the record itself says that most impacts are perpetual. The same reasoning made Larkfield’s triage service high.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A colleague proposes this measure: 'In how many of the ten records does our tier match the published level?' Why does the case study use another measure?
Show the answer
The level and the tier use different scales and different questions, so a match says little. The question that matters decides the work: which records get a close look before use? It was chosen before the scoring. With it, the baseline sends 0 of 10 to a close look, the method 6 and the method with the two questions 8.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A published register is an inventory that others can read. Canada’s has 38 records with results: 15 at level I, 23 at level II, none higher.
- Check a register before you use it: records without results, 5 questionnaire versions, one number under two labels, and answers that disagree with their points.
- A self-assessed level comes from the owner’s own answers. It is useful, and it is a claim to check.
- One measure, chosen first, made the comparison fair. The baseline gave a close look to 0 of ten records, the method to 6, the method with the two questions to 8.
- Larkfield should copy one record per system before use. It should add a named owner, a second scorer, results and what happened after launch.
Next, the final assignment: you put the course’s pieces together into Larkfield’s governance pack for its next year.
Next: Final assignment