Read the evidence pack

What you will be able to do

Find your way around the course’s files, and read every number with its label. A number without a label can come from a measurement, a price list or a guess. A leader who cannot tell them apart decides on the wrong one.

Before you startWhat to know and to have readyAbout 20 minutes

What you need to know

Nothing yet. This is the first lesson of the course. It also checks that you can read the course’s kind of evidence: a short report, a table of results and a simple cost estimate.

What you need

• A browser. The course’s files are on GitHub, in the folder C29 of the labs repository. You read them in the browser.
• A document tool: a word processor, a text editor or a spreadsheet. You copy templates into it and fill them in.
• No account, no installation, no code and no money.

To work offline, download the whole repository as a ZIP file: the green Code button, then Download ZIP. The course’s files are in the folder C29.

Four kinds of number in a car advert

Think of an advert for a used car. It gives four numbers, and each one means something different:

• “60,000 km on the clock.” Someone read it from the car. You can check it.
• “Price: 9,000 dollars.” The dealer published it. It can change next week.
• “The tyres should last another two years.” The dealer chose this number. A careful dealer says “one to three years, depending on how you drive”.
• A photo of a happy family on a road trip. Somebody made it up to show what the car could be like. It is not your family, and nobody says it is.

A careful buyer reads each number for what it is. She trusts the reading on the clock and checks the price on the day. She asks how the dealer chose two years, and she enjoys the photo without believing it.

The problem: one slide, three kinds of number mixed

Nadia joins Larkfield, an online home-and-garden shop in Canada, as the new head of customer operations. She leads Grace’s support team, and from now on she owns the decisions about its AI assistant. The assistant drafts a reply to each customer ticket. An agent reads and edits the draft before it is sent. The assistant’s model gets a written instruction with each ticket, the prompt. Today’s version is prompt v2.

On her first day, Omar, the product manager, shows her one slide. Nadia, Omar’s slide and this meeting are constructed for the course:

The support assistant saves US$287,661 a year. The new prompt v3 is cheaper and warmer: let’s switch.

Both sentences sound like facts. Nadia opens the evidence pack behind the slide. Most of it comes from a test in the course Evaluating and Testing AI Systems. There, the team ran both prompts on the same made-up tickets. Development tickets (102) are tickets that the team may study while it improves a prompt. Holdout tickets (98) are kept apart and read once, for the decision. Here is what Nadia finds:

• “US$287,661 a year” is calculated. It multiplies recorded results by numbers that somebody chose. The biggest chosen number, the minutes that an agent needs to write a reply by hand, was never measured.
• “Cheaper” was a guess. Prompt v3 sends a shorter request, so everyone expected it to cost less. The recorded test says the opposite: US$0.48 per 1,000 tickets against US$0.34 for today’s prompt v2.
• “Warmer” is true, and recorded. But the same test found that prompt v3 gave no answer at all on two safety tickets.

The slide is not a lie. Its problem is that every number looks equally solid. Nadia cannot see which numbers she can trust, which ones she can change, and which ones nobody measured.

The idea: every number carries a label

This course gives every number one of four labels, like the car advert. The labels tell you where a number comes from, and what you may do with it.

Label	What it means	Example from the course’s files
recorded	Measured in a run that an earlier course made and saved. The source names the course.	Prompt v2 sent 24 of 24 holdout tickets that needed a person to a person.
price	A published list price. Prices change: check the current page.	A provider’s list price to host a fine-tuned small model: US$1.70 an hour.
assumption	A value that someone chose, with a low, base and high value and a reason.	An agent needs 4 / 6 / 8 minutes to write a reply by hand.
constructed	A situation, a company or a document written for the course. Realistic, not real.	The vendor ReplyRight (made up for this course) and its offer; Nadia; Omar’s slide.

A fifth word appears often: computed, or “estimated”. A computed number is a recorded number multiplied by an assumption, by the course’s own calculation. It is only as strong as its weakest input, so the course always names the assumptions behind it. Omar’s US$287,661 is a computed number.

Why four labels, and not just “true” and “false”? Because each label asks for a different action. You can trust a recorded number, but only for what was tested. A price goes out of date. An assumption is an invitation to measure. A constructed example teaches, and proves nothing. The diagram shows the questions to ask about any number.

Four questions, asked in order, give every number its label. A number that fails all four has no source yet.

Text description of the diagram

The flow starts with one number in the pack and asks four questions in order. First: did a saved run measure it? If yes, the label is recorded, and you name the course that ran it. If no, the second question: is it a published list price? If yes, the label is price; you name the page and remember that prices change. If no, the third question: did someone choose it? If yes, it is an assumption, and you give its low, base and high values and the reason. If no, the last question: was the situation written for the course? If yes, it is constructed, and you say so every time. If no, the number has no source, and you do not use it until someone finds one.

A tiny example: label four numbers

Here are four numbers from the course’s files. Label each one before you read the answer under the table.

Number	Where it is
23 of 24	Holdout tickets that needed a person, which prompt v3 sent to a person
6 minutes	The base value of minutes_to_write_a_reply_by_hand in the value estimate
US$1.70 an hour	Hosting a fine-tuned small model
8 cards	The use-case cards, four at Larkfield and four in other fields

The first number is recorded. The course Evaluating and Testing AI Systems ran prompt v3 on 98 holdout tickets and saved every answer. The second is an assumption. Nobody timed Larkfield’s agents, so the course chose 6 minutes, with 4 and 8 as the low and high values. The third is a price from a provider’s list, used in the course Fine-Tuning and Adapting AI Models. The fourth is constructed. The cards describe made-up situations, except a few lines that are marked as recorded.

Models change; the method stays

Every recorded number in this course comes from one model on one day. A newer model can do better or worse on the same tickets. So read the recorded numbers as a snapshot, and keep the method. Use the same tickets, write the rule before the test, and read examples before scores.

The course’s files

The course has no code and no software to install. Its “project” is a folder of documents. The diagram shows the evidence pack of Larkfield’s support reply assistant, the folder that most lessons use.

Start with the one-page pilot summary. Each file behind it answers one question that a leader asks.

Text description of the diagram

On the left is the pilot summary: one page that says what was tested, the main results and what was not measured. Five arrows lead from it to the files behind it. File 1, evaluation results, answers “how good is it, and how sure are we?”. File 2, error examples, answers “what goes wrong, and what does it cost?”. File 3, review time and cost, answers “what is it worth, and what does review cost?”. File 4, operations, answers “what happened in a replayed week?”. The sources file answers “where does each number come from?”. Beside the pack are two more parts of the course’s folder. The templates are blank pages that you fill in. The use-case cards are eight ideas to screen. The phone version shows only the pack.

The folder also has a templates folder with one or more templates for each module, three case-study folders, and a README that lists everything. Every file is written for the course and is free to copy, also at work.

Check your files

This is the course’s first step. It checks that you can open every kind of file that the course uses.

1. Open the course’s README in your browser. Read its table “How to read the numbers”.
2. Open the pilot summary. Find how many holdout tickets that needed a person prompt v3 sent to a person. You see 23 of 24.
3. Open value-estimate.csv. GitHub shows it as a table. Find the row minutes_to_write_a_reply_by_hand. Its base value is 6, with the label assumption.
4. Open the use-case cards. Count the cards. You find 8.
5. Copy the value estimate template into your document tool. Check that its tables keep their rows and columns.

If you found all three figures, your files are complete. If a link does not open, download the ZIP file again and open the same file in the folder C29. You cannot break anything: these are copies. To start a template again, copy it again from GitHub.

Read a one-page pilot summary

Now read the pilot summary from the top. It is the one page that a busy leader reads first. Before you open the answer below, write two short lists in your own words: what the pilot measured, and what it did not measure.

Show one good answer

Measured (recorded): the answers of prompt v2 and prompt v3 on 271 made-up tickets, in English and French. That covers the team chosen, the “needs a person” decision, whether the reply follows the rules, the model cost and the time to an answer. Also a score for every reply from a judge model: a strong AI model that grades each reply against Grace’s policy. The course MLOps and LLMOps: Operating AI in Production replayed a practice week of tickets.

Not measured: real customers and real tickets; how long an agent needs to review a draft (an assumption); a second person’s labels. Also how agents and customers react, and whether agents trust the drafts too much.

The pack says this itself, in its table “What was measured, and what was not”. A good summary always has that table. When a summary has no such table, ask for it.

The summary has its own example of a number that changes meaning with its label. Prompt v3’s request is shorter: a median of 1,477 tokens against 1,902. A token is a piece of a word. On English text, one word was about 1.48 tokens, recorded in the course AI System Design and Cost Engineering. But with prompt v3, the model writes 643 tokens against 259, because it “thinks” longer before it answers. The provider charges for both. So the recorded cost is higher, and the guess “shorter means cheaper” was wrong.

Discuss: can Nadia quote US$287,661?

Nadia’s director asks her for one number for next year’s budget: what is the assistant worth?

• Omar says: “Quote US$287,661. It is in the evidence pack, so it is evidence.”
• Grace, the support lead, says: “Nobody has timed my agents. Six minutes per reply is a guess.”
• Kwame, the solution architect, says: “Quote the range, not one number.”

Before you read on, answer these:

1. Which label does US$287,661 have, and what is it made of?
2. Is Grace’s objection a reason to quote nothing?
3. What should the sentence in the budget say?

The course’s answer. US$287,661 is computed: recorded results times assumptions, and its biggest input was never measured. So it is the base case of a range, not a fact. The pack gives the range. The low case is a loss of US$52,905, and the high case is a gain of US$1,069,007. Grace’s point does not mean “quote nothing”. It means: say what the number rests on, and measure it. The budget line becomes: “Net value a year: a loss of US$52,905 to a gain of US$1,069,007, base US$287,661. It depends most on the minutes per reply by hand. We will time them before the next decision.” Estimate value and return builds this range step by step.

What would change the answer: a timed sample of agents writing replies. Then the minutes become a recorded number, and the range becomes much narrower.

When to use the labels, and what they cost

Use the labels on every number that goes into a decision: in a slide, a memo, a budget or a vendor’s offer. They cost a little time and some honesty. A number with a label often looks weaker than the same number without one. That is the point: the label shows what you still need to learn.

Do not stop at the label. A recorded number is true only for what was tested: 271 made-up tickets, one model, one day. An assumption with a range is better than a precise guess, but it is still a guess.

Common mistake

A number that lost its label

Symptom: Omar’s slide says “prompt v3 is cheaper”. Everyone in the meeting repeats it. Nobody can say where it came from.

Cause: an early expectation (“the request is shorter, so it costs less”) went into a slide without its label. After a few copies, it looked like a result.

Fix: open sources.md to see where each kind of number comes from, then find the cost in Evaluation results. The recorded model cost is US$0.48 per 1,000 tickets for prompt v3, against US$0.34 for prompt v2. If the pack has no source for a number, treat the number as a guess until someone finds one.

Consequence for a leader: a decision built on an unlabelled guess fails in a way that nobody can trace. Here, the “cheaper” prompt was dearer, slower and silent on two safety tickets.

Case studies in this course

At the end of the course, you use what you learn on a realistic project, from the data to the recommendation. Complete at least one for the certificate; the others are optional.

• Go/no-go memo for a support reply assistantReview a proposed AI feature with its pilot evidence (evaluation results, error examples, review time and cost), and write a go, no-go or go-with-conditions memo.
• Build, buy or partner for invoice extractionLead the decision as a finance domain expert: compare an in-house build, two vendor products and a partner on recorded trial results, cost ranges and data terms, and write the recommendation.
• Recover a stalled AI projectDiagnose a project that has run for six months without a launch (no definition of done, no evaluation set, no owner for adoption), and write a 90-day recovery plan with stage gates.

See it end to endGo/no-go memo for a support reply assistantCase study

In this case study, you use the whole pilot evidence pack for a real decision. Omar’s proposal shows only the numbers that help it; the pack’s labels show what was measured and what was assumed.

Knowledge check

Check your understanding

Label each number: (a) prompt v2 answered 98 of 98 holdout tickets; (b) Larkfield gets 600 / 822 / 1,100 tickets a day; (c) the vendor ReplyRight's offer; (d) US$287,661 net value a year.

Show the answer

(a) Recorded: the evaluation course measured it and saved the answers. (b) Assumption: a chosen range. Its base, 822, comes from the average day of a practice workload in Scaling APIs and AI Workloads. That workload is synthetic, so the course treats the volume as an assumption. (c) Constructed: the vendor and its offer are made up for the course. (d) Computed: recorded results multiplied by assumptions. It is an estimate, and you name the assumptions with it.

Check your understanding

Read the pilot summary. Name two things that the pilot did not measure, and say why each one matters to a leader.

Show the answer

Any two of these. Real customers and tickets: the 271 tickets are made up, so real tickets may be harder or easier. Review time: how long an agent needs to check a draft is an assumption, and it decides much of the value. A second person’s labels: one person labelled the right answers, so some “errors” may be disagreements. How people react: agents may trust the drafts too much and send wrong replies. Each gap is a risk to the decision, and each one can be measured later.

Check your understanding

Prompt v3 sends a shorter request than prompt v2, but its recorded cost per 1,000 tickets is higher. Explain why, in two sentences.

Show the answer

The provider charges for the tokens in the request and for the tokens that the model writes, including its hidden “thinking”. Prompt v3’s request is shorter (1,477 tokens against 1,902), but the model writes 643 tokens against 259. So the total is US$0.48 per 1,000 tickets against US$0.34.

Recap

• Every number in this course has a label. Recorded: measured in an earlier course’s saved run. Price: a published list price that can change. Assumption: a chosen value with a low, base and high range. Constructed: written for the course.
• A computed number is a recorded number times an assumption. It is only as strong as its weakest input, so you name the assumptions with it.
• The evidence pack starts with a one-page summary that says what was measured and what was not. Four files behind it answer four questions: how good, what goes wrong, what it is worth, what happened in use.
• Your files are complete when you find 23 of 24, the base value 6 and 8 cards.

Next, before any AI, you start from the problem: the user, the task and the cost of a mistake.

Next: Problems before tools