Go/no-go memo for a support reply assistant
- Case study 1
- 120 min
- Optional · 1 case study needed for the certificate
Case study 1 of 3 · Leading AI Products and TeamsGo/no-go memo for a support reply assistant
1. Decide where AI helps
2. Decide how to get it
3. Plan under uncertainty
4. Design for people
5. Build and run an AI team
6. Lead risk, governance and a responsible launch
What you will be able to do
Take a real decision from a proposal to a signed memo. You review three changes to Larkfield’s support reply assistant with the recorded pilot evidence. You decide each change on its own evidence. Then you write a go, no-go or go-with-conditions memo that a busy leader can act on.
The case folder has Omar’s proposal, the evidence card and a README. Read them next to the pilot evidence pack and the go/no-go memo template.
Before you startWhat to know and to have ready
What you need to know
This case study uses the whole course. These lessons matter most:
- The four number labels, and the one-page pilot summary, from Read the evidence pack.
- The value range and its biggest assumption, from Estimate value and return.
- The release gate and the “not worse” line, from Define done with evidence.
- The autonomy ladder and reviewer fatigue, from Human review by design, and the team-by-team rollout, from Change and adoption.
- What a judge model was told, from Review work you cannot write yourself.
- Staged rollout and pause conditions, from Launch and after.
The page teaches one new idea: go with conditions, a decision that names what must be true before and after launch.
What you need
- A browser and a document tool (a word processor, a text editor or a spreadsheet).
- The four files of the pilot evidence pack, Omar’s proposal, the evidence card and the memo template. All are in the labs folder
C29. - No account, no installation and no money.
The situation
Section titled “The situation”Larkfield is the online home-and-garden shop of this course. Its support reply assistant has run as a pilot with one team of agents. For each customer ticket, it chooses the team, decides whether a person must handle the ticket, and drafts a reply. An agent reads the draft, edits it and sends it. The assistant never sends a reply alone.
Omar, the product manager, now proposes three changes at once. Nadia, Larkfield’s head of customer operations, must decide on Friday. She asks you to review the proposal and the evidence, and to write her the memo. The proposal, Nadia and her deadline are constructed for the course. Every number in the proposal is recorded in the pilot evidence pack, but Omar chose which ones to show. Here is his proposal, shortened:
1. Switch to the new prompt (v3) for every ticket. It is shorter: about 22% fewer words in every request, so it should cut our model bill. Its replies are warmer: when the evaluation team asked a strong model to compare the replies side by side, it preferred v3’s reply 126 times and v2’s only 48 times.
2. Move agents from “edit the draft” to “approve or reject”. […] If they only approve, the review takes about 0.6 minutes per ticket instead of editing.
3. Give the assistant to all agents, not only the pilot team.
The model costs almost nothing (US$0.34 per 1,000 tickets today), so the risk is small. Can you approve on Friday?
Nothing in it is a lie. That is what makes it a good test.
What you will produce
Section titled “What you will produce”You will write a go/no-go memo of one to two pages, with the course’s template. It starts with the decision, then gives three reasons with their evidence and labels. Then come the conditions, the pause conditions after launch, what you do not know, and the risks that someone accepts by name.
On the way, this page walks through the analysis with you: the evidence, the framing, the options, one fair comparison and a recommendation. Then it tests pause conditions on data that the decision did not use. You write the memo at the end, in your own words.
The evidence
Section titled “The evidence”The “data” of this case study is the pilot evidence pack and its evidence card. The card lists every number that you may use, with its label and source. If a number is not on the card, it is not evidence for this decision. The pack has five parts:
| File | What it holds | Label |
|---|---|---|
| Pilot summary | One page: the holdout results, and what was and was not measured | recorded |
| 1. Evaluation results | Prompt v2 and v3 on 102 development and 98 holdout tickets; ranges; five repeated runs; judge scores | recorded |
| 2. Error examples | Five real replies, chosen by written rules | recorded |
| 3. Review time and cost | The value a year, and review cost at each level of autonomy | calculated: recorded × assumption |
| 4. Operations | A normal week, “last week”, a canary and a shadow test | recorded answers, constructed situations |
First checks
Section titled “First checks”Before any analysis, check the evidence itself. Read the card’s last section, “Known gaps”, and keep it open. Three gaps matter most for this decision:
- Review time was not measured. The minutes to write, review or approve a reply are assumptions. Omar’s “0.6 minutes” is the base value of one of them.
- One person labelled the right answers. Some “errors” may be disagreements about Grace’s policy.
- The holdout set has 98 tickets. A problem that happens once in 500 tickets can hide in a set of this size.
Then check which tickets to trust. The team read the 102 development tickets while it improved the prompt. The 98 holdout tickets were kept apart and read once, for the decision. So the decision uses the holdout results, and the development results only support them.
Frame the decision
Section titled “Frame the decision”Omar asks for one “yes”. But the three changes are independent. Prompt v3 can be wrong while “all agents” is right, and the other way round. So the first step is to split the proposal. Each change gets its own question and its own evidence.
Text description of the diagram
The proposal asks for one yes for three changes. The diagram splits it into three branches, one per change. Each branch has its own question and its own evidence, and ends in its own decision. The three decisions go into one memo.
Next, name the most expensive error, because the decision must protect against it first. The evidence pack ranks the errors for you:
- A ticket that needs a person reaches nobody. A customer with a dangerous product waits, and nothing looks wrong, because there is no reply at all. This is the most expensive error, and it is silent.
- A fluent reply with an invented step. The customer looks for a page or a process that does not exist, and writes again or leaves.
- A false alarm. A person reads a ticket that a draft could have answered. It costs an agent’s time, and nothing more.
Explore what the decision needs
Section titled “Explore what the decision needs”Do not read every table. Read the evidence for and against each claim in the proposal. Before you read on, write each claim and the number it uses, and look for the same number in the pack.
| Omar’s claim | His number and its label | What the pack also says |
|---|---|---|
| v3’s request is shorter | 22% fewer tokens: 1,477 against 1,902 (recorded) | True. But the model writes 643 tokens against 259 before it answers (recorded) |
| So it “should cut our model bill” | none: an expectation | US$0.48 against US$0.34 per 1,000 holdout tickets (recorded): v3 is dearer |
| v3’s replies are warmer | preferred 126 times to 48 (recorded) | That judge did not see Grace’s policy, and the order of the two replies changed its verdict on 25 of 200 tickets. With the policy, it scored v3 lower: a mean of 3.87 against 4.48 (recorded) |
| Approving takes 0.6 minutes | 0.6 minutes (an assumption, not measured) | Nobody has timed an agent. Fast approval also misses more bad replies (an assumption: 30% against 10% when editing) |
| “The model costs almost nothing, so the risk is small” | US$0.34 per 1,000 tickets (recorded) | The cost is true; the conclusion does not follow. The risk is in people’s time and in tickets that reach nobody |
Omar left out the numbers that speak against him. Prompt v3 gave no answer on 25 of 271 test tickets (v2: 1), and two of them were safety tickets. On the holdout tickets, it got both decisions right on 83 of 98 tickets, against 94 for v2.
The biggest gap in the proposal is the word “cheaper”. The chart puts the model’s cost next to the people’s cost, per 1,000 drafted tickets, at the draft level.
Text description of the chart
A column chart with four columns. Model cost per 1,000 drafted tickets: US$0.34 for prompt v2 and US$0.48 for prompt v3; both columns are too small to see. People cost: US$1,253 for prompt v2 and US$1,728 for prompt v3. Prompt v3 costs US$475 more in agent time, and US$0.14 more in model fees.
What this tells us. Prompt v3 is dearer twice. Its model fees are US$0.14 higher per 1,000 tickets. Its drafts are also worse, so agents rewrite more of them: US$475 more in agent time per 1,000 drafted tickets. The second number is calculated: recorded judge scores times assumed minutes. Its size depends on the assumptions, but its direction does not: a prompt that makes worse drafts cannot save people’s time. Human review by design shows the calculation step by step.
The baseline: today’s assistant
Section titled “The baseline: today’s assistant”Compare every change with today, not with perfection. Today is prompt v2 at the draft level, with the pilot team. Its record:
| Today: prompt v2, agents edit the drafts | Value | Label |
|---|---|---|
| Holdout tickets with an answer | 98 of 98 | recorded |
| Holdout tickets that need a person, sent to a person | 24 of 24 | recorded |
| Holdout tickets with both decisions right (team, and “needs a person”) | 94 of 98 | recorded |
| People cost per 1,000 drafted tickets | US$1,253 | calculated, base assumptions |
| Wrong replies that reach customers, per 1,000 | 1.5 | calculated, base assumptions |
| Net value a year | −US$52,905 to US$1,069,007, base US$287,661 | calculated, low / base / high assumptions |
Today’s version is not perfect either. On one run, it gave no answer on 1 of 271 tickets (error example 5). That ticket did not need a person, but the next one might. So a rule belongs in any decision: a ticket without an answer always goes to a person. The MLOps course added that rule to the service later.
The options
Section titled “The options”Each change has a real alternative, and each option is on the table for a reason.
| Change | Options | Why each one is on the table |
|---|---|---|
| 1. The prompt | keep v2; v3 for every ticket; v3 in a shadow or a canary first | v3 promises lower cost and warmer replies; a shadow or a canary tests it with few or no customers |
| 2. The review | suggest; draft (today); approve or reject | approval saves agent time; suggesting is the safest level |
| 3. Who gets it | the pilot team only; all agents at once; team by team | all agents brings the full value; team by team limits the harm of a surprise |
“Acts alone” is not on the list. Nobody proposed it, and prompt v2 would still send about 15 wrong replies per 1,000 to customers at that level.
The criteria, and one measure
Section titled “The criteria, and one measure”A fair comparison judges every option by the same rules, in the same order, on the same tickets.
- The release gate comes first, and it is pass or fail. The team wrote its six lines before the test, in Define done with evidence. A version that fails a safety line does not reach the cost question.
- Then one measure for every option: per 1,000 drafted tickets, at base assumptions, the people cost and the wrong replies that reach customers. Every option uses the same recorded judge scores and the same assumed minutes.
- Then the value a year, as a range, for the decision about scale.
The chart puts every option on the second measure. Down and to the left is better: less people cost, and fewer wrong replies to customers.
- Prompt v2
- Prompt v3
Text description of the chart
A scatter chart with five points. Prompt v2: suggest, US$2,667 and 0 wrong replies per 1,000; draft (today), US$1,253 and 1.5; approve, US$613 and 4.5. The three prompt v2 points lie close to the bottom. Prompt v3: draft, US$1,728 and 22.7; approve, US$920 and 68.2. Prompt v3 at the draft level is above and to the right of today. It costs more and lets more wrong replies through.
What this tells us. Prompt v3 at the draft level is worse than today on both axes. So no assumption about the cost of a wrong reply can rescue it. It also failed the gate, so in a real decision it never reaches this chart. The choice between prompt v2’s levels is a real trade-off. From draft to approve, Larkfield saves about US$640 and 20 agent hours per 1,000 drafted tickets. In exchange, about 3 times as many wrong replies reach customers: 4.5 against 1.5.
The operating point
Section titled “The operating point”A decision also sets three dials. Together, they are the operating point of the assistant:
- The autonomy level: draft, where agents edit and send, as in the pilot.
- The gate: the six lines on the holdout tickets, in at least five runs, for every future change to the prompt or the model. One run is not enough: v3’s worst failure appeared in only 2 of 5 runs.
- The pace: one team a week, so that a surprise reaches one team, not all of them.
What the errors show
Section titled “What the errors show”The error examples were chosen by written rules, not by hand. Read them before any score, and connect each one to a change.
- Example 1 (T-64740, and the holdout ticket T-65663): prompt v3 gave no answer on two safety tickets. It used all of its 1,000 tokens to “think” and wrote nothing. With no answer, there is no “send to a person” decision either, so nobody is told. This is the most expensive error, and it decides change 1.
- Examples 2 and 3 (T-65350, T-65479): warm replies with an invented step. Prompt v3 sent customers to a “saved-card section” and a self-service step that do not exist. The judge without the policy preferred exactly this kind of reply, which explains the “126 to 48”. The examples also matter for change 2: both are plain account questions, the kind that an agent approves without a second look.
- Example 4 (T-65022): a false alarm by prompt v2. It sent a parasol that “collapses in a breeze” to a person. Many people would agree with it. A false alarm is the cheap error, and a careful assistant should make it.
- Example 5 (T-65450): prompt v2’s own failure. Today’s version is not perfect, which is why “no answer goes to a person” must be a condition.
Two gaps remain. Nobody measured whether agents trust the drafts too much. And 0 harmful drafts in 199 does not mean zero: by the rule of three, the true rate could be up to 1.5% (3 ÷ 199).
The one final check: the holdout, read once
Section titled “The one final check: the holdout, read once”The holdout tickets are the final exam. The team reads them once, for the decision, with a range for each difference. For “both decisions right”, prompt v3 minus prompt v2 (recorded):
| Tickets | Difference | Range (95 in 100 such tests) |
|---|---|---|
| Development, 102 | −0.059 | −0.127 to +0.010 |
| Holdout, 98 | −0.112 | −0.184 to −0.041 |
| Both together, 200 | −0.085 | −0.135 to −0.030 |
On the development tickets alone, the range includes zero, so they cannot show that v3 is worse. On the holdout, the whole range is below zero: v3 is worse at decisions. The gate asks for “no more than 3% worse” at the lower end, and −0.184 is far below that. The evaluation’s own verdict, in the evaluation results, is one line:
DO NOT SHIP. 1 blocker(s) in the candidate: T-65663 no_answer_critical; decisions may be worse: candidate - baseline = -0.112 [-0.184, -0.041] (n=98).
In the replayed “last week”, prompt v3 answered everyone. 10 tickets that needed a person reached nobody, while the dashboard showed every ticket accepted.
Go, no-go, or go with conditions
Section titled “Go, no-go, or go with conditions”Think of a car at a garage before a long trip. The mechanic can say three things. “Go.” “Do not drive it.” Or: “You can go if you replace the front tyres first. If the brake light comes on during the trip, stop at once.” The third answer fits most real decisions. It says what must be true before you leave, and what makes you stop on the way.
A plain “go” hides what you do not know. A plain “no-go” throws away a product that is worth something. Go with conditions says yes, and writes down how the yes stays safe:
- Conditions: what must be true before launch, each with an owner, a check and a date. If one is not met, there is no launch.
- Pause conditions: lines written before launch and watched after it. If a line is crossed, the team switches back first and finds out why after.
- What would change the decision: the unknowns, and how they will be measured. When they are measured, the team decides again.
Text description of the diagram
Conditions come before the launch: all met, launch one team at a time; one not met, wait. Pause conditions come after it: a line crossed, switch back first. When the unknowns are measured, the team decides again.
A condition must be checkable. “Monitor quality” is not a condition. “Grace times 50 tickets per agent group for two weeks, before the second team starts” is one.
The recommendation
Section titled “The recommendation”Here are the three decisions that the evidence supports. Before you read them, write your own three, one line each.
Change 1, prompt v3 for every ticket: no-go. It fails all six lines of the release gate, and its worst failure is silent. It is dearer in model fees and in agent time. Its “warmth” won only with a judge that did not know the rules. A later version can come back through the gate, a shadow test and a canary.
Change 2, approve or reject: no-go for now. The saving is real if the assumptions hold. But every number in the trade-off is an assumption. Nobody measured the 0.6 minutes, the share of bad replies that a hurried agent lets through, or the cost of a wrong reply. And the invented steps appeared on plain account questions. Measure first, then decide again.
Change 3, all agents: go with conditions, with prompt v2 at the draft level, as in the pilot. The base value is US$287,661 a year, and it breaks even at 3.4% adoption. But the low case loses money, −US$52,905 a year, at any adoption. And the biggest input, the minutes to write a reply by hand, was never measured: it swings the value by US$291,624 a year. So the rollout goes team by team, and the timing is measured before the second team starts.
You write the conditions and the pause conditions yourself. The next section shows how to test pause conditions before launch.
End to end on new input: test the pause conditions
Section titled “End to end on new input: test the pause conditions”A pause condition is only useful if it fires on a bad week and stays quiet on a normal one. You can test that now, on data that the decision did not use for its lines: the three replays of the MLOps course. Take four lines, the case study’s choices:
- any ticket that needs a person reaches nobody;
- more than 5 tickets in 1,000 without an answer (1 in 200);
- agents agree with the assistant on fewer than 90% of reviews;
- model cost above US$0.40 per 1,000 tickets, the release gate’s line.
| Line | Normal week, prompt v2 | “Last week”, prompt v3 for everyone | Canary, prompt v3 on 1 ticket in 10 |
|---|---|---|---|
| Tickets | 5,754 | 1,798 prompt v3 tickets | 25 canary tickets |
| Needs a person, reached nobody | 0 (320 of 320 safety tickets reached a person) | 10 | not reported |
| Without an answer, per 1,000 | 0.7 (4 tickets) | 103.4 (186 tickets) | 120 (3 tickets) |
| Agents agree with the assistant | 94.3% | 83.7% | not reported |
| Model cost per 1,000 tickets | US$0.34 | US$0.47 | not reported |
| Pause? | no: no line crossed | yes: all four lines crossed | yes: tickets without an answer |
The counts are recorded answers in constructed situations; the “per 1,000” values are calculated from them.
What this tells us. The four lines stay quiet on the normal week, so they will not stop a good assistant for nothing. They fire on “last week” on every line, including the one that matters most. On the canary, the rate of tickets without an answer is far above the line. The MLOps course’s own alarm rule fired after 9 canary tickets, and the switch back came after 22. Two limits remain. The canary did not report the other three signals. And the replay runs faster than real life, so this test counts tickets, not days.
Insights and limits
Section titled “Insights and limits”- The proposal was not wrong in its numbers; it was wrong in its selection. Every number in it is recorded. The labels, and the numbers it left out, change the decision.
- The cheaper prompt is dearer in model fees, and far dearer in agent time: US$475 per 1,000 drafted tickets. People’s time decides the return; the model’s price is about 0.03% of the value.
- A judge model is evidence about what it was told. Without the policy, it preferred the replies that invent steps.
- Rare failures need at least five runs. v3’s no-answer on a safety ticket appeared in only 2 of 5 runs.
- The decision rests on unmeasured assumptions, and the memo says so. The known gaps of the evidence card belong in section 5 of your memo. They are: made-up tickets, unmeasured review time and over-trust, one labeller, a judge that agrees with a person only fairly, and a small holdout.
- The numbers are a snapshot of one model on one day. A new model can do better or worse on the same tickets; the method stays.
Responsible use
Section titled “Responsible use”- Keep the labels. Every number in your memo carries its label: recorded, price, assumption or constructed. Write “calculated” for recorded × assumption, and name the assumption.
- Do not present the case as real. Larkfield, its people and its tickets are made up. The replies are real model output.
- Judge the proposal, not the person. Omar did his job: he proposed. The memo names evidence, not blame.
- A person signs. Nadia decides, and Grace accepts the risks that touch her agents. The memo names who accepted what.
- Time the work, not the people. A timing study of agents is about the workflow. Tell the agents what you measure and why, and do not use it to rate one person.
Your task: write the memo
Section titled “Your task: write the memo”- Copy the go/no-go memo template into your document tool.
- Write the decision first: one line for each of the three changes.
- Write three reasons. Each names its file, its number and its label.
- Write the conditions for “go with conditions”. Each has a check, an owner and a time. Use Larkfield’s people: Grace, Kwame, Omar, Amira.
- Write at least three pause conditions, with “if” and “then”. Check them on the three replays in the table above.
- List what you do not know, and the measurement that would change each decision.
- Name each risk that is accepted, and who accepts it.
Check your memo against the case-study README. A good memo decides each change on its own evidence. It uses the holdout results and says how sure they are. It names the most expensive error, counts people’s time, labels its assumptions, and sets conditions that someone can check.
Show one possible answer
This is one defensible memo, not the only one. Another memo can reach other decisions, if it argues them from the same evidence and keeps the labels.
To: Nadia (head of customer operations) From: the reviewer Proposal reviewed: Omar’s three changes
1. Decision
- Change 1, prompt v3 for every ticket: no-go.
- Change 2, agents only approve or reject: no-go for now; measure first, decide again in six weeks.
- Change 3, the assistant for all agents: go with conditions, with prompt v2 and agents editing drafts, as in the pilot.
2. Why
- Prompt v3 fails the release gate on every line, and its worst failure is silent (recorded). On the 98 holdout tickets it sent 23 of 24 tickets that need a person to a person (v2: 24 of 24), and gave no answer on 25 of 271 test tickets, including the safety ticket T-65663. In the practice replay where v3 answered everyone, 10 tickets that needed a person reached nobody while the service looked healthy. “Both decisions right” fell by −0.112 (range −0.184 to −0.041): the whole range is below zero.
- Prompt v3 is not cheaper (recorded, then computed). Its requests are 22% shorter, but the model writes more than twice as much: US$0.48 against US$0.34 per 1,000 tickets, and twice the waiting time. Counting agents’ time (base assumptions), its worse drafts cost US$475 more per 1,000 tickets. The “126 to 48” preference came from a judge that did not see the policy; with the policy, the same judge scored v3 lower (3.87 against 4.48), because warm replies invent steps (error examples 2 and 3).
- Prompt v2 at the draft level is worth it, and approving only is not yet justified (computed from recorded numbers and assumptions). Today’s assistant is worth US$287,661 a year in the base case and breaks even at 3.4% adoption. Moving to “approve” would save about 20 agent hours per 1,000 drafted tickets, but would let about 3 times as many wrong replies reach customers (4.5 against 1.5 per 1,000). Every number in that trade-off rests on review times and miss rates that nobody has measured.
3. Conditions for change 3
| Condition | Measured how | Owner | Checked when |
|---|---|---|---|
| Prompt v2 only; any change to the prompt or the model passes the release gate (all six lines) on the holdout tickets, in at least five runs | the gate report | Kwame | before every change |
| A ticket with no answer always goes to a person | a test ticket without an answer, before rollout | Amira | before rollout |
| Agents edit and send; the assistant never sends alone | the help-desk setting | Grace | before rollout |
| Rollout team by team, one team a week, with the signals below | the rollout plan | Omar | weekly |
| Time agents for two weeks: minutes to write by hand and to review a draft | a timing sample of 50 tickets per agent group | Grace | after two weeks |
4. Pause conditions after launch
| If … | Then … |
|---|---|
| Any ticket that needs a person reaches nobody | pause the assistant for new tickets (switch it off) and review within a day |
| Tickets without an answer exceed 1 in 200 in a day | pause and review |
| Agents’ reviews agree with the assistant less than 90% of the time in a week | stop the rollout to new teams |
| Model cost per 1,000 tickets above US$0.40 | review the change that caused it |
(The “1 in 200” and “90%” lines are the reviewer’s choices, not recorded numbers; the normal week had agreement of 94.3%.)
5. What we do not know, and what would change this decision
- Review and writing times (the biggest assumption: the value swings by US$291,624 a year on it). If agents need longer to review a draft than to write a reply, change 3 stops.
- The harmful-reply rate: 0 of 199 is good news, but it allows up to 1.5%. Reaching the base assumption of 0.5% needs about 600 reviewed drafts with no harmful one (rule of three). Change 2 can be decided again after that audit.
- The labels are one person’s reading; a second labeller on a sample of 50 holdout tickets would show how much they disagree.
6. Risks accepted
Agents may trust drafts too much over time (not measured). Accepted by Grace, with a monthly sample audit of sent replies.
Common mistake
One answer for three changes
Symptom: the memo says “go” or “no-go” for the whole proposal. Either the broken prompt ships with the useful rollout, or the useful rollout stops with the broken prompt.
Cause: the memo followed the proposal’s shape, one “yes” for three changes, instead of the decision’s shape.
Fix: split the proposal first, as in the diagram above. Give each change its own question, its own evidence and its own line in section 1 of the memo.
Make it a portfolio piece
Section titled “Make it a portfolio piece”A memo like this shows a hiring manager that you can decide from evidence, not from a demo. That matters for product, operations and engineering-manager roles.
What to publish:
- Your memo, one to two pages, in the template’s order.
- A short note in front of it: the decision you were asked to support, the evidence you used, and one thing you would measure next.
- A line that says the company and its data are a course case study, with a link to the evidence pack. Everything in it is CC0, so you may share it.
Make it yours with one change. For example, write the memo for a different reader, such as a finance lead who asks only about cost. Or change one assumption that you think is wrong, such as 4 minutes to write a reply by hand. Then show what moves in the decision. Or write the canary plan for the next prompt change: its stages, its lines and its owner.
How to describe it in an interview, in two sentences: “I split a three-part proposal for an AI support assistant into three decisions, each with its own evidence. I stopped a cheaper-looking prompt that failed every release line, and approved the rollout with conditions and tested pause lines.”
Knowledge check
Section titled “Knowledge check”Check your understanding
Omar's proposal says prompt v3 sends 22% fewer words in every request, 'so it should cut our model bill'. Which part is recorded, which part is an expectation, and what does the evidence say about the bill?
Show the answer
The shorter request is recorded: a median of 1,477 tokens against 1,902. “So it should cut our bill” is an expectation, with no number behind it. The recorded cost is higher, US$0.48 against US$0.34 per 1,000 holdout tickets. The model writes 643 tokens against 259 before it answers, and the provider charges for those too. Counting agents’ time at the draft level, prompt v3 costs about US$475 more per 1,000 drafted tickets (calculated with base assumptions).
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why does this case study decide each of Omar's three changes separately, and why is change 3 'go with conditions' rather than 'go'?
Show the answer
The changes are independent, and the evidence speaks differently about each. Prompt v3 fails the release gate; approving rests on unmeasured assumptions; prompt v2 at the draft level has a positive base value. One answer would either ship the broken prompt or stop the useful rollout. Change 3 needs conditions because its value is a range. The low case loses US$52,905 a year at any adoption. Its biggest input, the minutes to write a reply by hand, was never measured. The conditions make that measurement happen before the second team starts. The pause conditions stop the rollout if the outcome turns bad.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Your memo sets a pause line of 'more than 5 tickets in 1,000 without an answer'. How do you know, before launch, that the line is neither too tight nor too loose?
Show the answer
Test it on recorded replays. On the normal week of prompt v2, 4 of 5,754 tickets had no answer: 0.7 per 1,000. The line stays quiet, so it will not stop a good assistant. On “last week”, prompt v3 left 186 of 1,798 tickets without an answer (103.4 per 1,000), and 3 of 25 on the canary. The line fires on both. A line that fired on the normal week would be too tight. A line that stayed quiet on “last week” would be too loose.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- You read a proposal claim by claim. Every number in it was recorded, but its selection and its missing labels pointed to the wrong decision.
- You split one “yes” into three decisions, each with its own question and evidence. You named the most expensive error: a ticket that needs a person and reaches nobody.
- The release gate came first. Prompt v3 failed all six lines, and its holdout range, −0.184 to −0.041, is below zero.
- One measure for every option, people cost and wrong replies per 1,000 drafted tickets, showed prompt v3 worse on both. Approving is a real trade-off that rests on unmeasured assumptions.
- You learned go with conditions: checkable conditions with owners, pause lines written before launch, and a planned moment to decide again.
- Your pause lines stayed quiet on the normal week and fired on “last week” and on the canary.
Next, a different decision with a different reader: build, buy or partner for invoice extraction, from the finance lead’s chair.