Recover a stalled AI project

What you will be able to do

Take over an AI project that has run for six months without a launch. You read its file, find the causes behind the symptoms, and decide what to keep and what to stop. Then you write a 90-day recovery plan with stage gates, and one paragraph that tells the owner: continue, change or stop.

Case studyPortfolio pieceSynthetic data

Dataset

Constructed project file (made for this case study)

Licence

CC0 1.0

Time

About 2 hours

Runs in

A browser and a document tool

Skills used

• Read a project file for missing decisions
• Symptoms and causes
• Baseline before model
• Money already spent
• Stage gates with stop results
• Owners for outcome and adoption
• Recovery plan

Open the case files on GitHub

The case files are documents, not code: read them in your browser, and write your answer in your own document tool. To work offline, download the repository as a ZIP file (the green Code button, then Download ZIP).

The case folder has three files: the case README, the project file and the evidence card. You fill in three templates from the same materials in your own document tool.

Before you startWhat to know and to have readyAbout 120 minutes

What you need to know

This case study uses the whole course. These lessons matter most:

• The four labels on every number, from Read the evidence pack.
• The use case, the baseline and the cheapest fix first, from Problems before tools.
• Experiments with a time limit, from Why AI plans are different.
• Stages, gates and stop results, from Milestones and roadmaps.
• The evaluation set, thresholds and one owner who signs, from Define done with evidence.
• Usage, adoption and outcome, from Change and adoption.
• The role map, from Team roles by stage.

What you need open

In the course’s practice materials:

• case-studies/recover-stalled-project/: README.md (the task), project-file.md (the evidence) and evidence-card.md (what each piece can show).
• templates/stage-gate-roadmap.md, templates/definition-of-done.md and templates/role-map.md, which you fill in.
• A document tool for your diagnosis and your plan.

The situation

Six months ago, Larkfield started a project to warn customers before a delivery is late. Victor, the head of operations at the time, sponsored it. He has since left the company. The budget is spent, the team says “nearly ready” every month, and nobody outside the team has used the product.

Nadia, Larkfield’s new head of customer operations, now owns the project. She asks you three questions. What went wrong? What should happen in the next 90 days? Should Larkfield continue, change the project or stop it?

The whole case is constructed for the course: the project, its documents, the people’s words and every number. It is built from the mistakes that this course teaches you to see. Larkfield’s team is the team of the other lessons. Three people are new: Victor (he), the former sponsor; Dana (she), a contract data engineer; and Luis (he), the delivery team lead.

What you will make

Three short documents, about one page each:

1. A diagnosis: three to five causes of the stall, each with its evidence from the file.
2. A 90-day recovery plan: owners, a stage-gate roadmap with a stop result at every gate, and a definition of done for the first gate.
3. A paragraph for Nadia: continue, change or stop, and why.

The page first works through the evidence with you. It does the first steps of each part and leaves the rest to you. Your task, with the full list of steps, is near the end.

The evidence and its labels

In the earlier case studies, many numbers are recorded from real runs. Here, every number is constructed. That is a deliberate choice: a real stalled project belongs to a real company, and its people would be named and blamed. A constructed project lets you study the pattern without that harm. So this case cannot tell you how often projects stall. It can teach you how to read one.

This table names every source of evidence that the page uses, with its label.

Source	What it holds	Label
project-file.md, section 1	The charter: goal, team, budget, “success”	constructed
project-file.md, section 2	Six monthly status reports by Tomás	constructed
project-file.md, section 3	What was spent: hours, a contract, services	constructed
project-file.md, sections 4 and 5	What six people say now; what exists today	constructed
evidence-card.md	What each piece can and cannot show; five known gaps	constructed
evidence/use-case-cards.md, card 4	The same problem, screened in Module 1: about 120 “Where is my order?” tickets a day	constructed
The labour rate	US$40 / 60 / 100 an hour (low / base / high), to turn hours into money	assumption (the system design course’s range)

The evidence card makes one warning that matters more than any number. The “accuracy” and “AUC” figures in the status reports cannot be compared: each month used a different test set or measure. Hold on to that as you read.

First checks

Before you analyse a file, check what it contains, as you would check a dataset. Read the project file once from start to end, and look for three things:

• Who is the user? The charter names a team and a sponsor. It names no user of the warning.
• What is the outcome? The goal says “reduce ‘Where is my order?’ tickets”. Nobody counted them, before or during the project.
• Who decides? The sponsor decided. He left in month 5, and nobody took his place.

Three empty answers, before any analysis. That is already a finding.

Frame the decision

Nadia has three choices, the same three as at every gate in this course: continue, change or stop. Two errors are possible, and they do not cost the same.

• Continue a project that cannot work. Larkfield pays for more months of work, and then a wrong warning reaches customers. A warning that says “late” when the parcel arrives on time is a broken promise. It may create the very ticket that it was meant to prevent.
• Stop a project that could have worked. Larkfield loses a cheap chance to cut tickets. But it keeps today’s way of working, which is slow but does no new harm.

The first error costs more, because it adds harm to waste. So the plan must find out early and cheaply whether anything works. It must also make “stop” a real, respected result.

Read the project file

Start with the status reports. Each month, Tomás reported a colour and a result. Here is what a reader of the reports saw, and what was happening underneath.

Six months of status reports (constructed). Five green months and one amber, while the test set changed, the measure changed, the data went stale and the sponsor left.

Text description of the diagram

The diagram has six columns, one per month. In each, the upper box shows the report. Months 1, 2, 3, 5 and 6 are green, and month 4 is amber. The results read: no score, accuracy 81%, accuracy 86%, accuracy 84%, AUC 0.88, AUC 0.91. They seem to rise. The lower box, in red, shows what happened underneath. In month 1, the pipeline was built, but no user was named. In month 2, Grace asked how customers would get the warning, and nobody answered. In month 3, the test set changed. In month 4, the carrier’s feed broke, so every later result used old data, and Dana’s contract ended. In month 5, the measure changed from accuracy to AUC, and Victor left. In month 6, the team asked for two more weeks, and Kwame says that it needs more than that.

Read the upper boxes alone and the project looks healthy: 5 green months out of 6, and a number that seems to rise. Read the red boxes too, and the numbers say almost nothing. The test set changed in month 3, so “accuracy 81%” and “accuracy 86%” are scores on different examples. The measure changed in month 5, so “accuracy 84%” and “AUC 0.88” measure different things. AUC measures how well a model ranks late deliveries above on-time ones; the machine learning course teaches it. It is not “accuracy, but better”. And from month 4, every result came from a copy of old data. Define done with evidence gave the rule: a set or a threshold that changes after the result measures nothing.

Now read section 3, what was spent. The file gives hours and amounts. To compare them, turn the hours into money with the course’s labour rate, an assumption. The chart shows the base case.

What the project spent: people's time at the base labour rate, against the services. Constructed hours and amounts; assumed rate.

Text description of the chart

Three bars. People’s time is 1,700 hours (constructed) at US$60 an hour (assumption): US$102,000. The weather data subscription is US$4,800, and the cloud and data services are US$2,900 (both constructed). The people bar is many times taller than the other two. With the low and high rates, people’s time is US$68,000 to US$170,000. Dana’s 3-month contract has no amount in the file, so it is not in the chart.

People’s time is about 93% of the known spending in the base case: US$109,700 in all, or US$75,700 to US$177,700 with the low and high rates. Estimate value and return showed the same pattern: people’s time, not the technology, decides the money. The weather data cost little. But it was paid for 12 months in month 3, and nobody has shown that it helps.

Last, read section 4, what people say now. Mark each statement as a fact that someone could check, or a view. Three of them matter most:

• Luis: “Half the late deliveries are late because a parcel is held at a depot. We know that the day it happens: the carrier’s status says ‘held’.” It is an expert’s estimate, not a measurement, as the evidence card says. But it is cheap to check, and it points at a rule.
• Priya: “I am not sure the model is better than a simple rule. We never agreed what ‘accurate enough’ means.” The person who did the evaluation says that no agreed evaluation exists.
• Omar: “It’s not on my roadmap. I thought operations owned it.” The owner in operations left, and nobody else took the project.

From symptoms to causes

Think of a car with a warning light on the dashboard. You can put tape over the light. The light is gone, but the fault is still there, and it comes back worse. A symptom is what you see: the light, “nearly ready”, a dashboard nobody opens. A cause is the decision that was missing or wrong, and that produced the symptom. A recovery plan that treats symptoms (“two more weeks to deploy”) stalls again. To find a cause, ask “why?” until the answer is a decision that someone should have made.

Ask 'why?' until you reach a decision that nobody made. Fix the cause, not the symptom.

Text description of the diagram

The chain starts with a symptom: the project is “nearly ready” every month. Why? Because no result could ever say “done”. Why? Because there was no agreed evaluation set, measure or threshold. Why? Because nobody owned the question “accurate enough for what?” Why? Because the charter’s goal was a technology, “an accurate model in production”, and not an outcome for a user. That is the cause. Fixing it means an outcome, an owner and a definition of done. A dashed arrow shows the other path: fix only the symptom with “two more weeks”, and the stall comes back.

Here is the first row of a diagnosis, worked from the chain. Keep the cause, the evidence and the symptom apart.

Cause	Evidence in the file	Symptom
The goal was a technology, not an outcome	Charter: “Use machine learning …”; success = “an accurate model in production”	“Nearly ready” every month; no count of “Where is my order?” tickets

Now do the same for the other symptoms. Three of them are in the course’s description of this case: no definition of done, no evaluation set, no owner for adoption. Look also at the “held” status that nobody tried, and at the green reports while the data went stale. Each one leads back, after a few “why?” questions, to a missing decision. Some symptoms share a cause; that is normal. Three to five causes are enough.

Common mistake

Blaming a person instead of a decision

Symptom: the diagnosis says “Tomás reported green when it was not” or “Victor left us without a plan”.

Cause: a person is the easiest thing to see in a story. But Tomás reported what nobody had asked him to measure, and people leave companies all the time. Both are consequences of decisions that nobody made.

Fix: write each cause as a missing decision or a missing role: “no owner of the outcome”, “status reports measured activity, not evidence”. Name people only as the source of evidence.

Consequence for a leader: a diagnosis that blames people makes the team defend itself, and the next status report hides problems. A diagnosis of decisions lets the same team fix them. The course MLOps and LLMOps: Operating AI in Production uses the same rule in its post-incident reviews.

The baseline nobody measured

Every option needs something to beat: the baseline, today’s way of working. Problems before tools screened this same use case, card 4, and stopped at the first question. A rule on the carrier’s status and the promised date could tell customers that a parcel is late. That screen was a short exercise. This project shows what happens when nobody does it.

The project has two baselines, and it measured neither:

1. Today’s way: no warning. Customers write a ticket, and an agent looks up the parcel. About 120 tickets a day, says Grace, but nobody counted them. Without a count, nobody can show that a warning reduces them.
2. The simplest fix: a rule. “If the carrier’s status says ‘held’, warn the customer the same day.” Luis says that this explains half the late deliveries. If he is right, the rule catches half of them from the first day, exactly, and anyone can check it on the tracking page.

A model must beat the rule by enough to pay for its upkeep. Nobody knows if it does, because the rule was never scored. So the first comparison in any plan is the model against the rule, on the same examples, with the same measure.

What is already spent

Nadia will hear an argument like this: “We have spent so much. Two more weeks, and it pays off.” The money is real, but it is gone, whatever she decides now. Economists call it a sunk cost: money or time spent in the past that no future choice can bring back. A good decision compares what each choice will cost from now on with what it can bring. So the 1,700 hours do not argue for continuing. They do not argue for stopping either.

The question that a sunk cost hides is “what still has value?”. Sort what exists today, from section 5 of the file:

What exists	Keep or stop?	Why
Six months of knowledge of the carrier’s data and statuses	Keep	It is in people’s heads, and the rule needs it
The data pipeline, broken since month 4	Keep, to repair	Any warning, from a rule or a model, needs live data
The dashboard prototype that nobody opened	Stop	It has no user and no owner
The weather data subscription	Do not renew yet	Nobody has shown that it helps

You finish this table in your task. The model notebook and “two weeks to production” are the hard rows: decide them with the evidence, not with the hours.

Three options under one measure

Nadia has three options. Each is on the table for a reason.

• A. Continue as planned: give Tomás the two weeks and connect the model to the website. On the table because the team believes in it, and the people who built it are still here.
• B. Change: reset the project with stage gates. Name the owners, count the problem, score the rule and the model on one fixed set, then decide. On the table because the file shows missing decisions, not proof that the idea is bad.
• C. Stop now. Close the project and keep today’s way of working. On the table because the budget is spent and nobody outside the team asked for the product.

Compare them under one measure: what each option costs from now until the next real decision, and what it teaches by then. That measure fits, because nobody yet knows whether a warning works. The best option finds out most cheaply, and can still stop.

	A. Continue	B. Change	C. Stop
Cost until the next decision	“Two weeks”, which Kwame says is too short; then an unknown amount	A time box that Nadia sets	Nothing more
What it teaches	Nothing about the outcome: no count, no baseline	Whether the problem is big, and whether the rule or the model works	Nothing
Risk to customers	High: an unchecked model on old data sends warnings	Low: no warning reaches a customer before a gate	None new; the tickets stay
Can it stop early?	No stop result exists	Yes: a stop result at every gate	It is the stop

Option A has no result that could ever stop it. Option C saves money but throws away a cheap chance to cut tickets. Option B costs a time box and gives Nadia evidence at each gate. It also keeps “stop” open, and a stop with evidence is better than a stop without it.

Discuss: continue, change or stop?

Nadia meets the team in her second week. She has read the file. These positions are constructed for the discussion.

• Tomás: “The model is good: AUC 0.91. Give me two weeks and it is live. Starting over wastes six months.”
• Priya: “I can’t defend any of our numbers. I’d rather score the rule and the model on one set first.”
• Kwame: “The pipeline has been broken since month 4. Two weeks is not realistic.”
• Grace: “I don’t care how it works. I want fewer ‘Where is my order?’ tickets, and I want to approve what customers read.”
• Luis: “Give us a rule on ‘held’ parcels next week. That’s half the problem.”
• Omar: “If it comes to my roadmap, I need an owner and a date.”

Before you read on, answer these:

1. Which positions are about the outcome, and which are about the technology?
2. Is Tomás’s “starting over wastes six months” a reason to continue?
3. What evidence would let Nadia choose between Tomás and Luis?
4. Omar wants a date. What can the plan promise instead?

The course’s answer. Change the project; do not stop it yet. Grace and Luis talk about the outcome; Tomás and Kwame talk about the technology. “Wastes six months” is the sunk cost: those months are spent either way, and what they taught stays. Only one test settles Tomás against Luis: the rule and the model, scored on the same frozen set of past orders, with one measure agreed first. Omar cannot have a launch date, because nobody knows the quality yet. He can have a date for each gate. This is the lesson of Why AI plans are different: an experiment with a time limit, then a decision.

What would change the answer: a count that shows few “Where is my order?” tickets. Then the problem is too small for any project, and stopping is right. Or a carrier that already tells customers when a parcel is held. Then the warning exists, and there is nothing to build.

The shape of a recovery plan

A recovery plan is a stage-gate roadmap, as in Milestones and roadmaps, with one extra stage at the start. A stalled project first needs a reset: owners named, the problem counted, the data repaired. Only then can feasibility ask a fair question. The diagram shows the shape. Your plan adds the time boxes, the evidence, the thresholds and the owners.

The shape of a 90-day recovery plan: a reset, then three stages, each ending at a gate with a written stop result. 'Ship the rule' is a success at gate 1.

Text description of the diagram

The flow starts with a reset stage: owners named and the problem counted. Gate 0 stops the project if the problem is too small. Next is feasibility: the rule and the model, scored on one frozen set with one measure. Gate 1 has three exits. If the rule is enough, ship the rule and stop the model; this exit is shown as a success. If neither works, stop. Otherwise, continue to the pilot, where warnings go to a share of orders and the team counts the tickets. Gate 2 stops or changes the work if tickets do not fall. Then a limited launch runs the warnings every day, with an owner. Gate 3 does not launch without an owner for the daily operation. Full launch comes after day 90.

Three parts of the shape answer the three gaps in the course’s description of this case:

• No owner for adoption: the reset names the owners. Use the role map: one person decides at every stage, and someone owns the outcome and the people whose work changes. Customers receive the warning, but Grace’s agents and Luis’s delivery team change how they work.
• No evaluation set: feasibility freezes one. Past orders with their real delivery dates are already in Larkfield’s data. Choose a period, freeze it, agree the measure, and then score every option on it.
• No definition of done: each gate gets thresholds, written in the definition of done before anyone reports a score. Measure what the warning is for, not “accuracy”. Change and adoption called it the outcome: here, fewer “Where is my order?” tickets.

For a warning, two measures fit the cost of a mistake. The first is the share of late deliveries that get a warning early enough. The second is the number of false warnings, because each one is a broken promise. You choose the thresholds, with a reason for each.

Test your plan on new input

A good plan decides what happens next. Before you hand it in, run it against two new events, both constructed. For each one, find the line of your plan that answers it.

1. Week 3. Tomás writes: “I tried another model type on a new test set, and the AUC is higher. Can we skip the comparison with the rule?” Your plan must answer without reading the number. The new set is not the frozen set, and AUC may not be the agreed measure. The comparison happens at gate 1, on its date, as written.
2. Day 60. Luis says that the carrier will change its feed format again next month. Your plan must name the person who owns the feed, and an alert for when its format changes. If no line covers it, your limited launch has the same gap that broke month 4.

If your plan cannot answer an event, add the line now. A gap found on paper costs less than a gap found at day 60.

Insights and limits

What the case teaches:

• The project did not fail at the model. It failed at the decisions around it: an outcome, an owner, a fixed test and a baseline.
• 5 green reports out of 6 hid every problem, because a status colour measures activity. A gate with a stop result measures evidence.
• The cheapest option, the rule, was in the file from the start, in Luis’s words. Nobody asked the delivery team.
• “Stop” can be a success. A plan that ends at gate 1 with “ship the rule” saves the model’s upkeep and cuts tickets sooner.

The limits, from the evidence card’s known gaps:

• Nobody measured the outcome. The 120 tickets a day is Grace’s figure, not a count. Your plan must count them before it can claim any change.
• The model’s real quality is unknown. The scores are not comparable, and every result after month 4 used old data. Do not quote them as evidence, for or against.
• Luis’s “half” is an estimate. It is the first thing to check, not a fact to plan on.
• The case is constructed. It shows a common pattern, not how often it happens. A real stalled project has more history, more politics and fewer clean quotes. The method still applies: find the missing decisions, then fix them with gates.

Responsible use

A recovery plan changes people’s work and customers’ messages. Four points keep it fair and safe:

• Diagnose decisions, not people. Quote people as evidence, and write causes as missing decisions or roles, as the failure case above shows. Share the diagnosis with the team before it reaches anyone else.
• A warning is a promise to a customer. A false “your parcel is late” is a broken promise, and a late warning is useless. Grace approves the wording. The pilot counts complaints about wrong warnings, not only tickets.
• Delivery data is personal data. Tracking records hold names, addresses and delivery times. The evaluation set needs dates and statuses, not addresses. Check what the pipeline keeps with the people who own privacy, as in Governance in practice.
• Stopping is a responsible result. If the evidence says stop, say so, and write it in a decision record. Do not keep a project alive to protect the hours already spent.

Your task

Work in your own document tool. Allow about 90 minutes.

1. Read the project file from start to end. Mark every place where a decision should have been made and was not.
2. Diagnose. Write three to five causes of the stall, each with its evidence from the file: a quote, a number or a missing document. Separate causes from symptoms. You may reuse the first row above.
3. Decide what to keep. Finish the keep-or-stop table. What still has value (data, people, a finding)? What is waste that Larkfield must stop paying for?
4. Write a 90-day recovery plan in the stage-gate roadmap. Include an owner who decides, a clear problem, a baseline that is not AI, and an agreed evaluation set. Add a definition of done for gate 1, an owner for adoption, and a stop result at every gate.
5. Test the plan on the two new events above, and add any line that is missing.
6. Write one paragraph for Nadia: continue, change or stop, and why.

What a good plan does (from the case README; reviewers use this list):

• It names a single decision owner and a domain owner: the people whose work changes.
• It compares the model with a simple rule or report before it spends more on the model.
• It fixes the evaluation set and the threshold before anyone reports a new score.
• It has gates with a time box and a result that stops the work, and “stop” is a real option.
• It plans adoption: who uses the warning, how customers receive it, and how success is measured. Success is fewer “Where is my order?” tickets, not “the model is accurate”.

Show one possible answer

This is one defensible answer, not the only one. It is the reviewers’ model answer, quoted unchanged. Everything in the case is constructed, and the thresholds are the reviewer’s example choices. A different decision is fine if you argue it from the same evidence, name your assumptions and meet the list above.

Diagnosis: five causes, with the evidence

Cause	Evidence in the project file	Symptom it produced
1. The goal was a technology, not an outcome	Charter: “Use machine learning …”; success = “an accurate model in production”	Nobody measured “Where is my order?” tickets, the reason for the project
2. No owner of the outcome or of adoption	Victor left in month 5; Omar: “not on my roadmap”; Luis: “nobody spoke to us”; Grace saw one demo	No plan for how a customer receives a warning; the dashboard that nobody opened
3. No agreed evaluation set, threshold or definition of done	Test set changed in month 3, measure changed in month 5; Priya: “the numbers are not comparable”	“Nearly ready” every month; the scores cannot be compared
4. No baseline	Luis: half the late deliveries show “held” the day it happens; nobody tried that rule	Six months of model work without knowing if a rule does as well
5. Status reports measured activity, not evidence	Month 4 “Amber” for a broken pipeline, then “Green” while the model runs on old data	The sponsor and the team believed it was on track

Keep, stop

• Keep: the knowledge of the carrier data, the pipeline code (to repair), the past orders with their real delivery dates (the start of an evaluation set), Priya and Luis’s insight about “held” parcels.
• Stop now: model work until Gate 1, the dashboard, the weather data renewal, the “two weeks to production” plan.

90-day plan

Decision owner: Nadia. Outcome owner: Grace (support). Process owner: Luis (delivery). Evaluation owner: Priya. Build: Tomás and Amira.

Stage	Days	Question	Evidence to pass	Stop or change if
0. Reset	1–10	What problem, for whom, how big?	“Where is my order?” tickets per 100 orders, counted for two weeks; the warning’s wording and channel agreed with Grace; pipeline repaired	The tickets are fewer than 20 a day: stop, the problem is too small
1. Feasibility	11–40	Does a warning catch late deliveries early enough, without too many false warnings?	One frozen evaluation set (the last 8 weeks of orders with real delivery dates); the “held” rule and the model scored on it with one agreed measure: share of late deliveries warned at least one day early, and false warnings per 100 orders	The rule meets the threshold: ship the rule, stop the model. Neither meets it: stop the project
2. Pilot	41–75	Do warnings reduce tickets for real customers?	Warnings sent for 10% of orders; tickets per 100 orders in the pilot group against the rest; complaints about wrong warnings	Tickets do not fall in the pilot group: stop or change the message
3. Limited launch	76–90	Can we run it every day?	An owner for the carrier feed and an alert when its format changes; a runbook; Grace’s sign-off	No owner for operations: do not launch

Example thresholds for Gate 1: at least 60% of late deliveries warned one day early, at most 2 false warnings per 100 orders. Gate 2: tickets per 100 orders 20% lower in the pilot group.

For Nadia

Change the project, do not stop it yet. Spend ten days to measure the problem and give it owners, and one month to compare a simple rule with the model on one fixed test. If the rule is enough, ship the rule and stop paying for the model. If nothing works, stop at day 40 with a clear reason: that is a good result, not a failure.

How to compare it with yours. Check that each of your causes is a decision, not a person, and has its evidence. Check that your plan scores the rule before the model, and that every gate has a stop result. One detail is open: this answer stops the weather data renewal, and you may prefer to test the weather data at gate 1 first. Either is fine with a reason.

Make it a portfolio piece

Recovering a stalled project is common work for product leads and engineering managers. A short, honest diagnosis shows an employer that you can see the pattern and act on it. Make it yours, then publish it.

What to publish (a page, a PDF or a repository):

• The diagnosis, one page: the causes, each with its evidence, and the keep-or-stop table.
• The 90-day plan, one page: owners, the stage-gate roadmap with stop results, and the definition of done for gate 1.
• The paragraph for Nadia.
• One line at the top: “The project is a constructed case from the course Leading AI Products and Teams (CC0).” Never present it as a real company’s project.

Make it yours with one change. Write the plan for a stalled project that you know, with names and numbers removed. Or change one fact in this case and show how the plan changes. For example, suppose that Luis’s count shows few parcels held at a depot.

How to describe it in an interview, in two sentences: “I diagnosed an AI project that ran six months without a launch. The stall came from four missing decisions: an outcome, an owner, a fixed test and a baseline. I wrote a 90-day recovery plan that scores a simple rule against the model first, with a stop result at every gate.”

Knowledge check

Check your understanding

The status reports show accuracy 81%, then accuracy 86%, then AUC 0.88, then AUC 0.91. Tomás says the model got better every month. What do you answer?

Show the answer

The numbers cannot show that. The test set changed in month 3, so “accuracy 81%” and “accuracy 86%” are scores on different examples. The measure changed in month 5, so the AUC figures measure something else. And from month 4, every result used old data, because the feed was broken. To say “better”, you need one frozen evaluation set and one measure, agreed before the scores, as in Define done with evidence. Until then, the real quality is unknown: neither good nor bad.

Check your understanding

Tomás says: "We have spent about 1,700 hours. Stopping now wastes them." Is this a reason to continue?

Show the answer

No. The hours are a sunk cost: they are spent whatever Nadia decides. A good decision compares what each option costs from now on with what it can bring. The hours are not a reason to stop either. The right question is what still has value. Here: the knowledge of the carrier data, the pipeline to repair, past orders for an evaluation set, and Luis’s insight. Keep those, whatever the decision.

Check your understanding

At gate 1, the 'held' rule meets the threshold, and the model does slightly better. What should the plan say, and why?

Show the answer

Ship the rule and stop the model, unless the plan set a margin that the model must beat. The rule is exact, cheap, easy to explain and easy to check on the tracking page. A model needs live data, upkeep and an owner, and it is sometimes wrong. A small gain does not pay for that. The plan should say this before the scores exist: “the model must beat the rule by at least …”. Then gate 1 needs no argument.

Recap

• A stalled project usually fails at missing decisions: an outcome, an owner, a fixed evaluation set and a baseline. The model is rarely the cause.
• Ask “why?” until you reach a decision that nobody made. Fix that, not the symptom. Write causes as decisions or roles, never as people.
• Scores on changing sets and measures cannot be compared. Freeze one set and one measure before any new score.
• Money already spent is a sunk cost. Decide on what each option costs from now on, and keep what still has value.
• A recovery plan is a stage-gate roadmap with a reset first. It scores the simple rule against the model, and “ship the rule” or “stop” are good results.

Next, the final assignment: an AI product and team plan for a use case of your own.

Next: Final assignment