Skip to content
Français

Review a published AI register

  • Case study 3
  • 120 min
  • Optional · 1 case study needed for the certificate

Where we are

In this course, we built Larkfield's inventory and tiers from its own system cards and registry. The registry said that the triage service was low, and the course's method said high. The Government of Canada publishes a register of this kind for everyone: 40 algorithmic impact assessments, each with an impact level. In this case study, we read that register as an inventory, tier ten of its systems with the course's method, and find what a published record leaves out.

What you will be able to do

Review a real, published register of automated decision systems. You build an inventory from public records, tier ten systems with the course’s method and compare your tiers with the published levels. Then you say what a published record leaves out, and what Larkfield should copy.

Case studyPortfolio piece

Time
About 2 hours
Runs in
A browser and a spreadsheet or document tool
Skills used
  • Read a register as an inventory
  • Check a dataset before you use it
  • Risk tiers with a written method
  • Self-assessed levels
  • One measure for a fair comparison
  • What a record leaves out
  • A two-page review

The case files are documents, not code: read them in your browser, or download them to work offline. Write your answer in your own document tool.

The case folder has a README and a source card. records.csv has one row per published record, and record-answers.csv has five answers that the course read from each one. The records themselves stay on the government’s portal.

Before you startWhat to know and to have readyAbout 120 minutes

What you need to know

This case study uses the whole course. These lessons matter most:

The page teaches one new idea: a self-assessed level.

What you need

Farida, Larkfield’s risk and compliance lead, has an inventory of eight systems and a tier for each one. Nadia, the executive sponsor, sends her a link with one question:

“The Government of Canada publishes a register of its automated decision systems. What can we learn from it for our own inventory? And what does a published register leave out?”

Larkfield, Nadia’s question and the team are constructed for this course. The records are not: they are real public records, and every count on this page comes from them.

The register exists because of a rule. The federal Directive on Automated Decision-Making applies to federal institutions, not to private companies (Treasury Board of Canada Secretariat, official source, checked 22 October 2026). Before an institution uses an automated decision system, it must complete and publish an Algorithmic Impact Assessment (AIA). The AIA gives the system an impact level from I to IV. So Larkfield has no duty here. It reads the register to learn from it.

You will make a two-page review: a ten-row inventory built from published records, a table “published level against our tier”, three findings and one recommendation for Larkfield.

Eight steps in two rows. 1, the records: 40 published, with the licence and the card. 2, first checks: 38 with results, 5 questionnaire versions. 3, explore the levels, scores and answers. 4, the baseline: take the published level at its word. 5, our reading of ten records with the course's method and two questions. 6, one measure: which records get a close look? 7, recommend for Larkfield's inventory. 8, new input: the newest record, then Larkfield's triage service.
The order of the review. The measure in step 6 is chosen before the results, so the results cannot choose it.
Text description of the diagram

Eight steps, in order. First, the records: 40 published, with their licence and the source card. Second, the first checks: 38 records have results, and they use 5 versions of the questionnaire. Third, explore the levels, the scores and the answers. Fourth, the baseline: take each published level at its word. Fifth, our reading of ten records with the course’s method and its two questions. Sixth, one measure: which records get a close look? Seventh, a recommendation for Larkfield’s inventory. Eighth, the method on new input: the newest record, then Larkfield’s triage service.

The portal’s collection “Algorithmic Impact Assessment” had 40 records when the course read it. Each record is a page on the Open Government Portal with a results document. The document has the department’s answers to a questionnaire, with points for each answer. It also has the impact level and the requirements of that level.

Before you count anything, check what the files cover:

  • 40 records, 38 with results. Of the other 2, one is the record of the questionnaire tool itself, and one has no PDF results document. Count 38, not 40.
  • 5 versions of the questionnaire. The column tool_version shows them. 26 records use version 0.10.0. The questions changed between versions, so the scores can mean different things.
  • One number, two labels. Most results documents call the raw score “Raw Impact Score”. 3 newer ones call it “Raw Score”. The course’s first script read only the first label, so it left 3 cells empty. An empty cell means “not found”, not 0, so the course opened those documents, found the numbers, and changed the script to read both labels. Now all 38 rows have a raw score.
  • One answer disagrees with its own points. The newest record answers “No” to “Will the system use personal information as input data?”, with 4 points. The other records give 4 points for “Yes” and 0 for “No”.

The course downloaded each results document again and compared its SHA-256 fingerprint with the pdf_sha256 column. All 38 matched, so the file describes the documents that are online.

What this tells us. The register is real and current, but it is not one clean table. It is 38 documents from 11 organisations, published from 2018 to 2026 with 5 versions of a form.

The impact levels of the 38 published records with results. No record is at level III or IV. Official source (Open Government Licence – Canada), counted by the course.
Text description of the chart

A column chart with four columns. Level I: 15 records. Level II: 23 records. Level III: 0. Level IV: 0.

What this tells us. Every published system is at level I or II, the two lowest of four. Yet the records include immigration decisions, benefits for veterans, passports and a hiring tool. Immigration, Refugees and Citizenship Canada publishes 13 of the 38. A register shows who publishes. It does not show who uses automated decisions and does not publish.

Each level comes from a current score. The records give it, but not where one level ends and the next begins.

The current score of each published record, by level and questionnaire version. In version 0.10.0, level I stops at 31 and level II starts at 34. In other versions, a score of 27 is level II. Official source, counted by the course.
Text description of the chart

Four rows of dots. Level I, version 0.10.0: scores from 25 to 31. Level II, version 0.10.0: from 34 to 63. Level I, other versions: 24. Level II, other versions: 27, 33, 34, 42, 43, 45, 47, 48, 49, 49, 56.

What this tells us. Within one version, the two levels do not overlap. Across versions, they do: a score of 27 is level II in an older record and level I in a version 0.10.0 record. So compare levels, not scores, and never compare scores across versions. In 10 records, the current score is lower than the raw score.

Three answers in record-answers.csv matter here:

  • Automation. “Full automation” in 2 records, “partial automation” in 25. 11 older records were not asked. Both fully automated systems are at level I.
  • Peer review. Each record prints the requirements of its own level. All 15 records at level I say “Peer review: None”. All 23 at level II list experts to consult.
  • A responsible executive. Only 3 records, all in version 1.0.1, have a field for the assistant deputy minister responsible for the program.

Think of buying a house. The seller fills in a form about it: any water damage, any old wiring? The form is useful, but it is the seller’s form. A buyer still pays an inspector, who looks at the house and writes what was found.

A self-assessed level is a level that comes from the owner’s own answers. The answers have points. The points give a score, the score gives a level, and the level decides the requirements. Nobody outside scores the system before the level exists. The level is useful: it is fast, the same for everyone and public. But it measures the answers, not the system. Larkfield’s registry tier is the same kind of thing: the triage team gave its own system “low”.

A flow. The department's own answers, each with points, give the raw impact score. Mitigation answers, such as consultation, data checks and recourse, join it to give the current score, which is lower than the raw score in 10 of 38 records. The current score gives the impact level, I to IV. The record prints the requirements of its level: at level I, peer review none; at level II, peer review. A dotted line from the level leads to a box: no field for measured results, complaints, a review date, or whether the system is still in use.
How a published level is made. Every input is the department's own answer. The record has no field for what happened after the system went live.
Text description of the diagram

The department answers the questionnaire, and each answer has points. The impact answers give the raw impact score. The mitigation answers, such as consultation, data checks and recourse, join it to give the current score. In 10 of 38 records, the current score is lower than the raw score. The current score gives the impact level, from I to IV. The record then prints the requirements of its level: no peer review at level I, and peer review at level II. A dotted line leads to what the record has no field for: measured results, complaints, a review date, and whether the system is still in use.

So read a published level as the seller’s form: a good start, and a claim to check. As Evidence and audits said, a description is not evidence.

The simplest review copies the published level into the course’s tiers. Level I is low, level II is medium, and levels III and IV are high.

On all 38 records, this baseline gives no high tier at all, because no record is above level II. If Farida worked this way, the hiring tool and the passport system would get the light checks of a meeting-notes tool.

What this tells us. The baseline is cheap. But a review that only copies the level adds nothing: the level is already published.

Two other ways to review a record:

  1. The course’s method. Score the four factors from the record’s own answers: harm, people affected, autonomy and data. Then apply the two rules. Score what the system does alone, not what a person checks later, as for Larkfield’s triage service.
  2. The method and the two questions. First ask the method’s two questions. Is the system’s area one that a law calls high-risk? The EU AI Act lists eight such areas in its Annex III (Regulation (EU) 2024/1689, official source, checked 22 October 2026). Examples are employment, migration and border control, and essential public services. And does it decide about a person with no person involved? A yes sends the record to the specialist first.

The EU law does not apply to the Government of Canada. Here, the first question only sorts records. It is not a legal finding about any of them.

The course chose ten records: five at level I and five at level II, easy cases and hard ones. Their factor scores are the course’s reading of each record’s answers, not the department’s view.

One measure: which records get a close look?

Section titled “One measure: which records get a close look?”

The level and the tier use different scales, so “do they agree?” is a weak question. A better question decides the work: which records get a close look before use? A close look means a high tier, or the specialist first. The measure was chosen before the scoring.

Record (short name)Published levelOur factors (harm, people, autonomy, data)Our tierThe two questionsClose look: baseline, method, method and questions
Staffing: automated interview scoresI3 + 3 + 2 + 2 = 10highemploymentno, yes, yes
Old Age Security: absences to investigateI2 + 3 + 1 + 2 = 8mediumpublic benefitsno, no, yes
Police: transcription of recordingsI2 + 2 + 1 + 3 = 8mediumlaw enforcementno, no, yes
Social Insurance Number: automated issueI2 + 3 + 3 + 3 = 11highdecides aloneno, yes, yes
Border: export compliance dashboardI1 + 2 + 1 + 2 = 6lownono, no, no
Access-to-information request websiteII1 + 2 + 2 + 1 = 6 (one score unclear)lownono, no, no
Immigration: spouse eligibility approvalsII2 + 3 + 3 + 2 = 10highmigration and border controlno, yes, yes
Veterans: mental health benefitII3 + 3 + 3 + 3 = 12highpublic benefits; decides aloneno, yes, yes
Veterans: claim summary toolII2 + 3 + 1 + 3 = 9highpublic benefitsno, yes, yes
Passports: facial recognitionII3 + 3 + 2 + 3 = 11highbiometricsno, yes, yes
Close look before use Baseline (the level as is) The course’s method The method and the two questions
Of the ten records 0 6 8

What this tells us. The baseline sends none of the ten to a close look. The method sends 6, and the two questions add 2 more. Our tier is higher than the baseline for 8 records, the same for 1 and lower for 1. So the method does not raise everything. The access-to-information website is level II, and our reading is low: 1 + 2 + 2 + 1 = 6.

Two records explain most of the gap. The Social Insurance Number system answers “full automation” and is level I. It decides alone about a number that people need for work: 2 + 3 + 3 + 3 = 11, high. The spouse eligibility model approves eligibility alone, and officers make the final decision later. It never refuses. The department scored it level II; we score what it does alone: 2 + 3 + 3 + 2 = 10, high.

Farida shows the table at the team meeting. The staffing record is at level I: it scores candidates’ recorded interviews automatically.

  • Tomás: “The government assessed it and published it. If they say level I, it is level I.”
  • Mei: “Its own record checks ‘Employment’ in the list of decision areas. That adds one point.”
  • Omar: “And in this pilot, an HR advisor reviews every candidate. So a person decides.”
  • Sam: “Then why does our method say high?”

Before you read on, answer these:

  1. Score the four factors from the record. Which rule or question decides the tier?
  2. What does level I require, according to the record itself?
  3. What would you need to see before you accept level I?

The course’s answer. By our reading, harm is 3 and people affected is 3: the result touches access to a job. Autonomy is 2: it scores every candidate, and a person reviews. Data is 2: recorded interviews. The sum is 3 + 3 + 2 + 2 = 10, high. Before scoring, the first question already says yes: employment is an Annex III area. So in Larkfield’s process, this tool goes to the specialist first.

The record prints the requirements of level I: “Peer review: None”. Omar is right that a person reviews every candidate in the pilot, but that is an answer in a form. To accept level I, you need evidence. That means test results against human raters and across groups of candidates, and a date for the next review. The record describes these tests as planned, with no numbers.

What would change the answer: suppose the scores only scheduled interviews and decided nothing about who stays in the pool. Then harm would drop to 2 and autonomy to 1: 2 + 3 + 1 + 2 = 8, medium. The first question would still send it to the specialist.

Copy the register’s good habits. One record per system. The record exists and is published, inside Larkfield, before the system is used. It keeps the answers that gave the tier, not only the tier. A questionnaire is quick and the same for every team.

Add what it leaves out. Every row needs four more things:

  1. A named accountable owner. Only 3 of 38 records have a field for the responsible executive.
  2. A second scorer for any self-assessed tier. The risk lead scores with the team, as in Risk tiers and proportionate controls.
  3. Results, not plans: the last evaluation, with its date.
  4. What happened after launch: complaints, corrections and the next review date.

A tier without evidence is an opinion, even when it is published.

The newest record in the file, published 14 August 2026, is the traveller compliance indicator of the Canada Border Services Agency. It is at level II, and you have not used it yet. Run the whole review on it:

  1. Checks. It uses version 0.10.0 of the questionnaire, with a raw score. It is the record whose personal-information answer disagrees with its points. Write that down as a question for the department, not as a fact.
  2. Inventory row. Purpose: indicators that an officer uses to release a traveller or refer them to a second examination. The record answers that its clients are “particularly vulnerable”. No accountable owner field, no review date.
  3. The two questions. Border control is an Annex III area: specialist first.
  4. The method. 2 + 3 + 1 + 2 = 8: medium. Autonomy is 1, because an officer decides.
  5. Close look. The baseline says no, the method alone says no, and the two questions say yes.

What this tells us. On new input, the review still finds what the level alone hides, and a gap in the record itself.

Now turn the review on Larkfield. Its registry is a register too, with tiers that each team gave itself. The registry says that the ticket triage service is low. The method says 3 + 2 + 3 + 2 = 10, high. It is the only registered system where the two disagree. That is the same finding as the hiring tool: a self-assessed level, with no second scorer and no evidence attached.

Insights.

  • No published record is above level II, although the systems touch immigration, benefits, passports and hiring.
  • A published level is a score of the department’s own answers. It measures the form, not the system.
  • One measure, chosen first, made the comparison fair: a close look for 0, 6 or 8 of ten records.
  • Larkfield’s registry has the same weakness, and the same fix: a second scorer and evidence.

Limits.

  • The factor scores are our reading of public answers. A department knows its system better; a different reader may score one point differently.
  • Ten records of 38 are a sample chosen by the course, not a random one.
  • The register shows only what was published. It cannot show systems without a record, or say why none is above level II.
  • The portal changes. Your copy of the file is the course’s extract; the live collection may have more records.
  • Review records, not people. The records name the public servants who completed them. Never copy those names; the licence does not cover personal information.
  • Do not present our tier as the government’s. Say “the course’s reading of the published answers”, every time.
  • No legal conclusions. The EU AI Act does not apply to these systems. The Annex III question only sorts them for a specialist. Whether a system meets the Directive is for the institution and its reviewers.
  • Give the acknowledgement from the source card wherever you use the records.

Write your own answer before you open the one below.

  1. Inventory. Fill ten rows of the inventory template from published records: the course’s ten, or ten of your choice. Mark each field as filled from the record, partly filled or missing.
  2. Tiers. Ask the two questions, then score the four factors with one reason each, from the record’s own answers. Mark any score that is unclear.
  3. Compare under one measure. For each record: close look or not, by the baseline, the method, and the method with the two questions.
  4. Three findings about the register as a whole, each with a count from the files.
  5. One recommendation for Larkfield’s inventory, and what would change it.
Show one possible answer

This is one defensible answer, not the only one. A different score is fine when you give the record’s answer that supports it.

To: Nadia. From: Farida. Subject: what Canada’s published AI register teaches our inventory.

The register. 38 of 40 records have results (Open Government Licence – Canada). Level I 15, level II 23, level III 0, level IV 0. 11 organisations; 13 records from one department.

Inventory: what a record fills. Name, purpose, data and autonomy: filled from the answers. Kind (built or bought), people affected, vendor, status (only the project phase) and tier (a level on another scale): partly. Accountable owner (except 3 records), technical owner and review dates: missing.

Ten records, published level and our tier:

RecordLevelOur factorsOur tierClose look?
Staffing: automated interview scoresI3 / 3 / 2 / 2highyes
Old Age Security: absences to investigateI2 / 3 / 1 / 2mediumyes
Police: transcription of recordingsI2 / 2 / 1 / 3mediumyes
Social Insurance Number: automated issueI2 / 3 / 3 / 3highyes
Border: export compliance dashboardI1 / 2 / 1 / 2lowno
Access-to-information request websiteII1 / 2 / 2 / 1lowno
Immigration: spouse eligibility approvalsII2 / 3 / 3 / 2highyes
Veterans: mental health benefitII3 / 3 / 3 / 3highyes
Veterans: claim summary toolII2 / 3 / 1 / 3highyes
Passports: facial recognitionII3 / 3 / 2 / 3highyes

Findings.

  1. No record is above level II, and the 2 fully automated systems are at level I. The baseline gives a close look to 0 of 10; our method and the two questions give it to 8.
  2. The level is self-assessed. At level I, the record’s own requirements say “Peer review: None” (15 of 15). No record has a field for measured results, complaints or a review date.
  3. The register is uneven. 5 questionnaire versions from 2018 to 2026, scores that cannot be compared across versions, two labels for the same score and one answer that disagrees with its points.

Recommendation. Keep the register’s habit: one record per system, written before use, with the answers that gave the tier. Add four fields to every row: a named accountable owner, a second scorer, the last evaluation result with its date, and complaints and corrections since launch. Apply it first to the triage service, the one system whose registered tier the method contradicts.

What would change it. If the records began to report results and reviews, our second scorer could use them, and reviews would be faster.

This case study shows an employer that you can review a real register with a written method, and say what it cannot show. Risk, compliance, audit and public-sector teams look for this.

What to publish:

  • Your ten-row inventory, with each field marked filled, partly filled or missing.
  • The table “published level against our tier”, with the close-look measure.
  • Three findings with their counts, and the recommendation.
  • The acknowledgement: “Contains information licensed under the Open Government Licence – Canada.” Say that the tiers are your reading with a course method.

How to describe it in an interview, in two sentences: “I reviewed Canada’s 38 published algorithmic impact assessments as an inventory, and tiered ten of them with a written method. None is above level II, so I showed which systems a self-assessed level would let through without a close look, and what each record should add.”

Go further:

  • Run your review on the UK’s algorithmic transparency records, published under the Open Government Licence. What do they have that the Canadian records do not?
  • Open the live collection. Are there records newer than 14 August 2026? Review one.

Check your understanding

Every published record is at level I or II. Give two reasons why that does not prove that the government's automated decision systems are low-impact.

Show the answer

First, the level is self-assessed: it is a score of the department’s own answers, and the record has no field for measured results, complaints or reviews. Second, the register shows only published records. It cannot show systems without a record. A third reason: the questionnaire changed over 5 versions, so the scores are not on one scale.

Check your understanding

The spouse eligibility record says that officers make the final decision on every application. Why does the course still score its autonomy 3?

Show the answer

Score what the system does alone, not what a person checks later. The model approves eligibility by itself; the officer then screens for admissibility and decides. A wrong approval at the eligibility step is rarely overturned: the record itself says that most impacts are perpetual. The same reasoning made Larkfield’s triage service high.

Check your understanding

A colleague proposes this measure: 'In how many of the ten records does our tier match the published level?' Why does the case study use another measure?

Show the answer

The level and the tier use different scales and different questions, so a match says little. The question that matters decides the work: which records get a close look before use? It was chosen before the scoring. With it, the baseline sends 0 of 10 to a close look, the method 6 and the method with the two questions 8.

Recap

  • A published register is an inventory that others can read. Canada’s has 38 records with results: 15 at level I, 23 at level II, none higher.
  • Check a register before you use it: records without results, 5 questionnaire versions, one number under two labels, and answers that disagree with their points.
  • A self-assessed level comes from the owner’s own answers. It is useful, and it is a claim to check.
  • One measure, chosen first, made the comparison fair. The baseline gave a close look to 0 of ten records, the method to 6, the method with the two questions to 8.
  • Larkfield should copy one record per system before use. It should add a named owner, a second scorer, results and what happened after launch.

Next, the final assignment: you put the course’s pieces together into Larkfield’s governance pack for its next year.

Next: Final assignment