Skip to content
Français

Incidents, complaints and redress

  • Lesson 18
  • 35 min
  • Module 6: Run governance

Where we are

In the last lesson, we set up a board for the changes that people plan. Incidents are not planned: a wrong result has already reached a customer, and sometimes the customer complains. In this lesson, we handle one AI incident and one complaint, the way a good restaurant handles a dish that should never have left the kitchen.

What you will be able to do

When an AI result harms someone, decide quickly who acts and put the result right for every person affected. Then change the system so that it does not happen again.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need

A dish that should never have left the kitchen

Section titled “A dish that should never have left the kitchen”

A guest tells the waiter about a nut allergy. The dish arrives with nuts in the sauce, and the guest feels ill.

A good manager acts in order. She helps the guest, takes charge and tells the truth. The restaurant pays for the meal and the doctor. She checks the other tables that ordered the same sauce, and asks whether the health inspector must be told. Then she changes the kitchen: allergy orders get a red ticket, and the chef checks each one.

A bad manager says sorry, offers a free dessert and changes nothing. Next month, it happens to another guest.

Larkfield’s support-resolution workflow proposes actions, and a team lead approves each one. In week 3 of its pilot, this happened (incident INC-07, constructed; you first met it in From policy to control):

Incident INC-07 as a flow. The workflow proposes actions in week 3 of its pilot. A team lead approves: 212 approvals, 0 opened, median 3 seconds. 212 actions are carried out. One German ticket gets a replacement shipped, although the policy says that a person decides. The customer gets a replacement and no answer to the question. A dashed line: Grace's weekly sample of 10 approved actions finds it.
Incident INC-07 (constructed): every approval was a click. Only Grace's weekly sample of ten actions found the one that broke the policy.
Text description of the diagram

In week 3 of the pilot, a team lead approved 212 proposed actions and opened 0 of them first. The median time to approve was 3 seconds. One German-language ticket about a damaged item got a replacement shipped, although the policy sends this kind of ticket to a person. The customer got a replacement and no answer to the question. Grace found it in her weekly sample of ten approved actions.

The situation is constructed, but the behaviour is recorded. In AI Agents and Workflow Orchestration, every proposal was approved as it was. Then one design made 3 changes that nobody asked for, in 2 of 70 tickets. With a strict approver, it made 0. Recorded numbers like these come from one model on one test set. They show what the records look like, not how good a model is today.

Now look at how INC-07 was found. Grace samples 10 of about 212 actions a week. If only one action is wrong, a random sample of 10 contains it with a chance of 10 ÷ 212 = 4.7% (calculated). So Larkfield found it by luck. And then nobody knew who should act: the team lead, Amira, who built the workflow, or Grace.

The idea: incidents, complaints and redress

Section titled “The idea: incidents, complaints and redress”

An AI incident is an event in which an AI system’s result harmed someone, could have, or broke a rule. A complaint is a person telling you that a result about them is wrong. It is often how you learn of an incident.

Redress means putting a wrong right for the person it affected. You correct the result, explain it and repair the harm, and they can ask for a review by a person. The word comes from an old French word for “set straight again”.

Every incident and complaint goes through five questions, in order:

Five questions after an incident is found by a sample, a complaint, a monitor or a member of staff. 1 Who decides? The accountable owner, who stops the harm first; a serious incident goes to the board within 2 working days. 2 What do we tell the people affected? What happened, what we do, and how to reach a person. 3 Correct the result: for this person, then search for others like them. 4 Must anyone outside be told? Ask the lawyer, and write the answer and the reason in the record. 5 What changes? Design, measure, training; the board confirms.
Five questions for every incident and complaint. Question 3 does not stop at one customer, and question 4 always gets a written answer.
Text description of the diagram

After an incident is found: 1, who decides? The accountable owner, who stops the harm first; a serious incident goes to the board within 2 working days. 2, what do we tell the people affected: what happened, what we do, how to reach a person. 3, correct the result for this person, then search for others. 4, must anyone outside be told? Ask the lawyer and record the answer and its reason. 5, what changes: design, measure or training, confirmed by the board.

Question 4 needs care. Whether a report outside is required depends on the law, the place and what happened, for example whether personal data was exposed. This course does not list those duties, and it is not legal advice. So the record always has a line for question 4, with the lawyer’s answer and its reason. For a bought tool, the vendor’s promise about incident notice matters too: TicketHarbor answered “without undue delay”, MinuteMoss “within 72 hours” (both made up for this course).

A worked example: INC-07, questions 1 to 4

Section titled “A worked example: INC-07, questions 1 to 4”

Farida and Grace take INC-07 through the questions. These are the intermediate results, before the board meets:

Question INC-07
1. Who decides? Grace, the accountable owner of the workflow. She stops the click-through approvals in the pilot queue the same day. One customer is affected and no data was exposed, so it waits for the next board meeting, not an extra one.
2. What do we tell the customer? The truth, within one working day: a replacement was sent, but nobody answered the question. Here is the answer, and the customer may keep the replacement. Grace’s team lead writes it.
3. Correct the result For this customer: the answer. For others: Grace’s team checks the week’s 212 approved actions for other tickets that the policy sends to a person.
4. Must anyone outside be told? Farida asks the lawyer, and writes the answer and the reason in the record.
5. What changes? For the board to decide. You write it in the module practice.

Look at question 3. The team lead approved 212 actions without reading them, so the one that Grace found may not be the only one.

The cause is not “a careless team lead”. Nobody can read 212 proposals a week with care, and the screen let a click count as a review. MLOps and LLMOps: Operating AI in Production shows how to run that review in its lesson Post-incident learning. Governance owns the questions around it: who decides, who is told and what changes.

A customer in Quebec asked for a refund for a late delivery. The reply said: “Our system determined that you are not eligible for a refund.” She asks who decided, what information was used, and for a person to review the decision (complaint CMP-14, constructed). The workflow drafted the reply, and a team lead approved it in 4 seconds. The review found that the parcel was 9 days late. Grace’s policy refunds the delivery fee after 7 days late.

  • Omar: “A person approved it, so it was not an automated decision. We have no special duty here.”
  • Tomás: “It talks about rights. Send it to the lawyer and wait for an answer.”
  • Grace: “Refund the US$12.95 and close the ticket.”
  • Sam: “What did she ask for?”

Before you read on, answer these:

  1. What is wrong with each of the first three positions?
  2. What must the reply contain?
  3. What exactly do you ask the lawyer?

The course’s answer. Sam asks the right question. The customer asked who decided, what information was used, and for a review by a person. Larkfield answers all three, whatever the lawyer says. A person on Grace’s team reviews the decision and finds it wrong. The reply refunds the US$12.95 delivery fee, says that the order’s delivery dates were used, and names a person she can reply to.

Omar’s view is the legal question, not its answer. You met it in Canada, the US, the UK and others. For a decision based exclusively on automated processing, Quebec’s section 12.1 asks the business to tell the person. On request, it explains the information used and the main reasons, and the person can submit observations to a staff member who can review the decision (Quebec’s private-sector privacy act, section 12.1, official source, checked 22 October 2026). For customers in the EU, the GDPR gives a right not to be subject to some decisions based solely on automated processing: those with legal or similarly significant effects (GDPR, Article 22, official source, checked 22 October 2026).

So Farida asks the lawyer two questions. “Is an approval in 4 seconds a real human review under these rules? And what must we tell customers about replies that the workflow drafts?” Tomás is right to ask, and wrong to wait: the customer gets her answer now. Grace’s refund corrects one result. Other late parcels were probably refused the same way, so Grace’s team reviews every refused refund for a delivery more than 7 days late.

What would change the answer: if refusals already went to Grace’s team with the policy rule on the screen, CMP-14 would be one person’s mistake, not a design problem.

When to use the five questions, and what they cost

Section titled “When to use the five questions, and what they cost”

Use them for every incident and complaint about an AI result. A small one takes the owner ten minutes and a short record, which the board reads in item 1.

The control that worked was the cheapest: ten actions a week, read by the owner. It is still weak, as the 4.7% chance shows. So keep it, and add controls that do not depend on luck. The screen shows the action and the policy rule, and nobody can approve a ticket that the policy sends to a person.

Common mistake

The apology that changes nothing

Symptom: the same kind of complaint comes back next month from another customer.

Cause: the team corrected the result for the one person who complained. Nobody searched for others with the same problem, and nothing in the design changed.

Fix: answer questions 3 and 5 every time: search for the others before you close it, and write the change, its owner and its measure.

  1. Open your agenda from the last lesson and the minutes table of the board’s agenda.

  2. Write the minutes line for INC-07: the decision, the reason, the conditions with an owner, and what would change it.

  3. Read request R-12 with INC-07 in mind.

    Check: the same workflow, and the same approval step that failed in the pilot.

  4. Write the minutes line for R-12.

  5. Read TicketHarbor’s answers against the vendor questionnaire.

  6. Write the minutes line for R-04.

    Check: each condition has a named owner, and each decision says what would change it.

Show one possible answer
Item Decision Reason Conditions (owner) What would change it
INC-07 Stop the click-through approvals in the pilot queue; redesign the approval 212 approvals, 0 opened, median 3 seconds: the control was not working The approval screen shows the exact action and the policy rule; a team lead cannot approve a ticket that the policy sends to a person (Amira). The weekly sample stays (Grace). A reply to the customer’s real question within 1 day (Grace). New measure: approvals opened before approving (Farida) A second case in the sample
R-12 Go with conditions, as a 4-week pilot on one queue The tier goes from medium to high. The harm is money and can be corrected. In the agents course, the router design (the model picks one path, and code runs its fixed steps) recorded 0 changes nobody asked for A limit and a daily total, and only the router design (Amira); a weekly sample of 20 and a pause on one forbidden refund (Grace); INC-07’s fixes first (Farida) A forbidden refund in the sample; the approval fixes not done
R-04 Keep it off until the contract changes; then switch it on Training is on by default, and the sub-processors are not named Contract terms (Grace and the finance lead); agents read every suggestion, with training (Farida) The vendor refuses the terms: look for another vendor

Check your understanding

The incident record for INC-07 says: 'Cause: the team lead did not read the proposals.' Why is this a weak finding, and what should it say instead?

Show the answer

It blames a person, so its only fix is “read more carefully”, which fails again next week. Nobody reads 212 proposals a week with care. A useful finding names the design. The screen did not show the action and the policy rule. And nothing stopped the approval of a ticket that the policy sends to a person. The team can change and measure those.

Check your understanding

Predict: Grace doubles her weekly sample from 10 to 20 actions out of about 212. If one action in the week is wrong, what is the chance that the sample finds it? Is that enough?

Show the answer

About 20 ÷ 212, or 9.4%: twice as good, and still mostly luck. A sample shows whether approvals work in general; it does not catch each rare error. So add a control that blocks the error by design: a team lead cannot approve a ticket that the policy sends to a person.

Check your understanding

Write the three things that Larkfield's reply to the customer in CMP-14 must contain, and the question that Farida asks the lawyer.

Show the answer

The reply says three things. A person reviewed the decision, found it wrong and refunded the US$12.95 delivery fee. The information used was the order’s delivery dates. And she can reply to a named person. The questions for the lawyer: is an approval in 4 seconds a real human review under Quebec’s section 12.1, and the GDPR’s Article 22 for EU customers? What must Larkfield tell customers about replies that the workflow drafts? The course does not answer that question itself.

Recap

  • An AI incident is a result that harmed someone, could have, or broke a rule. A complaint is often how you learn of one. Redress puts the wrong right for the person affected.
  • Five questions, in order: who decides (the accountable owner, who stops the harm first), what to tell the people affected, how to correct the result for them and for others, whether anyone outside must be told (the lawyer answers, the record keeps the reason), and what changes.
  • Find the cause in the design, not in a person.

You can now decide on planned changes and handle what goes wrong. But does all this make Larkfield safer, or does it only add forms and waiting? Nobody can tell yet. Next, you measure governance itself.

Next: Measure governance