Final assignment

What you will be able to do

Use every module of the course on one new prediction problem, and write your reasoning so that another person can check it. The result is an illustrated reasoning workbook: numbers, diagrams and plain-language conclusions that match the data.

Before you startWhat to know and to have readyAbout 150 minutes

• You completed Modules 1 to 6. Each part below names the lessons that it uses.
• Make a new file or document with the name c03-workbook. A Markdown file, a word-processor document or paper are all good. Hand-drawn diagrams are good too: take a clear photo of each one and put it in the workbook.
• Have a calculator. All the data that you need is on this page. You do not need code. If you like, you can check your calculations with Python, but you must show the values in the workbook.

The problem: late deliveries at Larkfield

Many tickets at Larkfield say “Where is my order?”. The delivery team has an idea: if a model can predict which orders will arrive late, an agent can contact those customers before they write. Omar, the product manager, asks you for a short workbook. He wants to know what the data says, how the model learns, whether a new delivery route is really better, and which threshold the team should use.

Larkfield promises delivery in 5 days. An order is late when it takes more than 5 days.

All the data below is fictional.

Part 1: describe the delivery times

Uses Mathematical reading skills, Describe a dataset and Read distributions.

Delivery time in days for 15 orders from one week:

Order	O-1	O-2	O-3	O-4	O-5	O-6	O-7	O-8	O-9	O-10	O-11	O-12	O-13	O-14	O-15
Days	2	2	3	3	3	3	4	4	4	5	5	6	7	9	14

1. Calculate the mean, the median, the range and the standard deviation. Write the unit with each value.
2. Calculate the share of late orders, as a fraction and as a percentage.
3. Draw a dot plot or a histogram of the 15 values. Mark the mean and the median on it.
4. Write two or three sentences: which summary should Omar quote for a “normal” delivery, and what does that summary hide?

Part 2: find similar past orders

Uses Represent observations, Vector operations and Similarity and distance.

An agent wants to see past orders that are similar to a new order, N. Each order is a vector of two features: distance in kilometres and parcel weight in kilograms.

Order	Distance (km)	Weight (kg)	Arrived
N (new)	420	2.0	?
P1	400	18.0	on time
P2	150	2.5	on time
P3	450	2.2	late

1. Write the shape of the data for the three past orders as (rows × columns), and say what one row and one column mean.
2. Calculate the Euclidean distance from N to P1, P2 and P3 with the raw values. Which past order is nearest?
3. Scale both features: divide each distance by 500 km and each weight by 20 kg. Calculate the three distances again. Which past order is nearest now?
4. Draw the four orders in two dimensions, once with raw values or once scaled (your choice), and explain the difference in two sentences.
5. Say which result you trust for “similar delivery”, and why.

Part 3: what a “late” flag means

Uses Events and uncertainty, Conditional probability and Distributions in practice.

Over a long period, 6 of 100 orders arrive late. The model flags 80 of 100 late orders. It also flags 10 of 100 orders that arrive on time.

1. Use 1,000 orders. Draw a frequency tree: late or on time, then flagged or not flagged. Write a count on every branch.
2. Make the 2 × 2 table of counts from the tree.
3. Calculate the share of flagged orders that really arrive late.
4. Write one sentence that explains to an agent what a flag means. Do not use the words “80% accurate”.
5. The team handles 50 orders an hour. With a late rate of 6%, how many late orders do you expect in one hour, and is it surprising to see 6 in one hour? Explain with the binomial idea, in words.

Part 4: trace a model as it learns

Uses Functions and change, Loss and gradients and Optimization behaviour.

A first, very simple model predicts the extra days of an order from its distance: $\text{extra days} = w \times \text{distance}$, with distance in hundreds of kilometres. Three past orders:

Distance (hundreds of km) x	1	2	3
Extra days y	1	3	4

With these three orders, the mean squared error and its gradient are:

$$L(w) = \frac{26 - 38w + 14w^2}{3} \qquad \frac{dL}{dw} = \frac{28w - 38}{3}$$

1. Say in words what the unit of w is.
2. Calculate the loss at w = 0, w = 1 and w = 2. Draw the loss curve through these points.
3. Start at w = 0 with a learning rate of 0.1. Trace three updates in a table: w, the loss, the gradient and the new w.
4. Start again at w = 0 with a learning rate of 0.25. Trace three updates.
5. Mark both sets of steps on your loss curve. Explain in two or three sentences why one rate works and the other does not.

Part 5: is the new route better?

Uses Samples and populations, Estimate uncertainty and Compare alternatives.

The delivery team tried a new route plan. In one week, with the old plan, 28 of 400 orders were late. In the next week, with the new plan, 22 of 400 orders were late. Tomás writes: “The new plan reduces late deliveries by more than 20%.”

1. Calculate both late rates, and the difference in percentage points. Then check Tomás’s “more than 20%”: what is it 20% of?
2. Calculate an approximate 95% confidence interval for the difference. Use this standard error: $\sqrt{\frac{p_1(1-p_1)}{400} + \frac{p_2(1-p_2)}{400}}$.
3. Draw an interval plot: the two late rates with their intervals, or the difference with its interval and a line at zero.
4. The two weeks were different weeks. Name two other reasons why the second week can have fewer late orders.
5. Write your conclusion for Omar in three parts: the observed effect, the uncertainty, and what the data can and cannot say about cause. Suggest a better test.

Part 6: recommend a threshold

Uses Classification trade-offs, Thresholds and costs and Calibration and slices.

On a test set of 1,000 orders, 60 were late. The model gives each order a score from 0 to 1. At three thresholds:

Threshold	Orders flagged	Late orders among the flagged
0.2	300	57
0.4	142	48
0.6	60	33

Costs: an agent contacts every flagged order, which costs 1 cost point. A late order that nobody contacted costs 12 cost points (an angry customer and a “where is my order” ticket).

1. For each threshold, make the confusion matrix and calculate precision, recall and the false-positive rate.
2. Calculate the total cost at each threshold. Draw a chart of cost by threshold.
3. Recommend a threshold, and explain the trade-off in plain words.
4. Change one condition: a late order without contact now costs 25 cost points. Calculate again. Does your recommendation change?
5. A slice check at threshold 0.4: city orders, 35 late, 31 flagged; rural orders, 25 late, 17 flagged. Calculate the recall for each slice and say what it means for your recommendation.
6. List at least three pieces of evidence that you need before the team uses the model with real customers.

Part 7: one page for Omar

Write a summary at the start of the workbook, at most one page:

• Three findings, each with its number and its unit.
• One uncertainty that Omar must know about.
• Your threshold recommendation and its condition (“if the cost of a late order is about … “).
• One claim that the data does not support.

Template

# Workbook: late deliveries at Larkfield

## Summary for Omar (one page)

## 1. Delivery times
Mean, median, range, SD (with units):
Late share:
Diagram:
Which summary, and what it hides:

## 2. Similar past orders
Shape:
Raw distances:            Scaled distances:
Diagram:
Which result I trust, and why:

## 3. What a flag means
Frequency tree:
2 × 2 table:
Share of flags that are late:
Sentence for agents:
Late orders in one hour:

## 4. The model learns
Unit of w:
Loss at 0, 1, 2:
Trace at rate 0.1:        Trace at rate 0.25:
Diagram:
Why one rate works:

## 5. The new route
Rates and difference:
Interval:
Diagram:
Other reasons:
Conclusion (effect, uncertainty, cause) and a better test:

## 6. Threshold
Confusion matrices and metrics:
Costs and chart:
Recommendation:
With cost 25:
Slices:
Evidence needed before use:

Check your work

There is no reviewer for this course yet. Review your workbook with this table, or ask a colleague to use it. Mark each item as demonstrated, partially demonstrated or not yet demonstrated, with a reason. The course is complete when every item is demonstrated. You can correct an item and review it again as often as you need.

Required evidence	Demonstrated when
Correct units and calculations	Every number has its unit (days, km, kg, percentage points, cost points). The calculations in all six parts are correct, and every intermediate value is visible, so a reader can find a mistake. Percent and percentage points are not mixed.
Clear diagrams	Each part has its diagram: a dot plot or histogram with mean and median, a vector drawing, a frequency tree, a loss curve with steps, an interval plot, a cost chart. Each diagram has labelled axes or branches, units, and a one-line caption that says what to see.
Uncertainty explained honestly	The workbook says what the 400-order weeks can and cannot show, gives the interval and does not call the new route “better” without it. The share of late flagged orders is not confused with the model’s recall. Small samples, such as the slices, are named as small.
Statistical conclusions matched to the data	Each conclusion follows from the numbers in the workbook. The summary for Omar has no claim that the data does not support, in particular no cause from the two-week comparison. The threshold recommendation states its cost condition and the evidence still needed.

Correct numbers with a wrong conclusion are not enough. For example, if the workbook calculates the interval correctly but still says “the new route reduces late deliveries”, the last item is not yet demonstrated.

Show the key numbers, to check your calculations

Open this only after you have finished your own calculations.

• Part 1: mean ≈ 4.93 days, median 4 days, range 12 days (2 to 14), SD ≈ 3.04 days (population) or ≈ 3.15 days (sample). Late: 4 of 15 ≈ 26.7%.
• Part 2: raw distances: P1 ≈ 25.6, P2 = 270.0, P3 ≈ 30.0 (P1 is nearest). Scaled: P1 ≈ 0.80, P2 ≈ 0.54, P3 ≈ 0.06 (P3 is nearest).
• Part 3: 60 late (48 flagged, 12 not), 940 on time (94 flagged, 846 not). 48 of 142 flags are late ≈ 34%. Expected late orders in an hour: 50 × 0.06 = 3.
• Part 4: L(0) ≈ 8.67, L(1) ≈ 0.67, L(2) = 2.0. Rate 0.1: w = 0 → 1.267 → 1.351 → 1.357. Rate 0.25: w = 0 → 3.167 → −1.056 → 4.574 (diverges). The best w is 19/14 ≈ 1.357.
• Part 5: 7.0% and 5.5%, difference 1.5 points (21% of 7.0%). Standard error ≈ 1.71 points; 95% interval ≈ −1.9 to +4.9 points.
• Part 6: cost 336, 286 and 384 cost points at 0.2, 0.4 and 0.6. With 25 points: 375, 442 and 735. Slice recall at 0.4: city ≈ 89%, rural 68%.

AI assistants

You can ask an AI assistant to check one calculation or to explain a step again. The numbers, the diagrams and the conclusions must be yours. Check every number an assistant gives you with your calculator: assistants make arithmetic mistakes and can sound certain when they are wrong. That is the lesson of this course in one sentence.

Common mistake

Every number is correct, and the conclusion is too strong

Symptom: the workbook has correct means, distances, intervals and costs. The summary for Omar says: “The new route reduces late deliveries by 21%, and the model catches 80% of late orders, so agents can trust the flags.”

Cause: the summary was written from memory of the claims, not from the numbers in the workbook. The interval includes zero, and the weeks were different. And 80% is the recall: only about one flag in three is a late order.

Fix: write the summary last. For each sentence, point to the part and the number that supports it. If no number supports it, remove the sentence or say what evidence is needed.

Recap

In this course, you learned to read the numbers of AI work and to make decisions with them:

• Summaries: the mean, the median, the spread and a chart, and what each one hides.
• Vectors: observations as lists of numbers, the dot product as a score, and similarity that depends on scaling.
• Probability: counts first, the base rate always, and the difference between “flag given late” and “late given flag”.
• Learning: a loss to make smaller, a gradient that points uphill, and a learning rate that decides if training works.
• Evidence: samples, intervals and effect sizes, and what a comparison can and cannot say about cause.
• Decisions: a threshold chosen by costs, and checks of calibration and slices before you trust a score.

Keep your workbook. The next courses use these ideas every day: the SQL course prepares data, and Machine Learning: From Problem to Reliable Model trains and evaluates machine learning models.