Skip to content
Français

From forecast to order and roster

  • Lesson 24
  • 45 min
  • Module 7: Uncertainty and the decision

Where we are

We have quantile forecasts that we can trust. Now we make the decision. In this lesson, we use the newsvendor idea to choose the order quantile from the cost of a lost sale and a left-over item, staff the help desk the same way, and compare the cost of each method.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Turn a forecast range into the order and the roster with the lowest expected cost, and compare forecasting methods by the cost of their decisions.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It covers the parasol’s best order, two ordering rules over the backtest, and the roster. Its setup cells install the packages and download the Larkfield data, so it needs no local setup. About 45 minutes, with about 1 minute of waiting.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

What you need on your computer

Nothing: the notebook runs in Colab or Kaggle. To run it in VS Code instead, use the folder and the environment from Meet the Larkfield data.

Tested with Python 3.12, pandas 2.3.3, statsforecast 2.1.1 and SciPy on macOS. The Larkfield data is synthetic: we made it for this course.

A boy sells newspapers at a street corner. He buys them for 1 dollar each in the morning and sells them for 3 dollars. In the evening, today’s paper is worth nothing. How many does he buy?

If he buys too few, every customer after he runs out is a lost profit of 2 dollars. Every extra paper is a loss of 1 dollar. So he buys a little more than he expects to sell, but not a lot more.

This is the newsvendor problem, one of the oldest problems in planning. Kenji has the same one every week.

The problem: the forecast ignores the costs

Section titled “The problem: the forecast ignores the costs”

When Kenji ordered the forecast added up, half of the orders ran short. The middle of the range treats a lost sale and a left-over item as if they cost the same. At Larkfield, they do not (the course’s assumptions):

  • A lost sale costs the margin: 35% of the price. For a parasol at 79.99 dollars, that is 28.00 dollars.
  • A left-over item is sold at a 60% discount at the end of the season. Larkfield bought it at 65% of the price and sells it at 40%, so it loses 25% of the price: 20.00 dollars for a parasol.

So Kenji should order above the middle. How far above? “The 90% quantile, to be safe” sounds careful, but every unsold extra parasol costs 20 dollars.

Think about the last parasol of the order. Should Kenji add one more?

  • If demand reaches that parasol, it sells, and it saves a lost sale: +28 dollars.
  • If demand does not reach it, it is left over: −20 dollars.

Say the chance that demand reaches it is p. On average, the extra parasol is worth p × 28 − (1 − p) × 20 dollars. That is positive while p is larger than 20 ÷ 48 = 0.417. So Kenji stops where demand is above the order 41.7% of the time, and below it 58.3% of the time. That is the 58.3% quantile of demand.

The general rule is the critical ratio:

best quantile = cost of a lost sale ÷ (cost of a lost sale + cost of a left-over item)

For Larkfield: 0.35 ÷ (0.35 + 0.25) = 0.583. The price cancels out, so the same quantile works for every product. The newspaper seller buys at 2 ÷ (2 + 1) = 0.67.

A tiny example: three orders for the parasol

Section titled “A tiny example: three orders for the parasol”

On Sunday 8 June 2025, ETS forecast 92.4 parasols for the next 14 days, with a standard deviation of 20.4 for the total. For a bell-shaped range like this, a formula gives the expected units short and left over for any order. The course’s forecasting job (Module 8) uses it too. Three orders:

Order Quantile Expected short Expected left over Expected cost Stock-out chance
93 50% (the middle) 7.8 8.4 388 dollars 49%
97 58% 6.0 10.6 382 dollars 41%
119 90% (“to be safe”) 0.9 27.5 576 dollars 10%

The expected cost is the units short × 28 dollars plus the units left over × 20 dollars. The 58% order is the cheapest, but only by 6 dollars: the bottom of the cost curve is flat. Ordering “to be safe” costs 194 dollars more. It saves about 5 lost sales (6.0 → 0.9) and adds about 17 left-over parasols (10.6 → 27.5).

Larkfield really sold 101 parasols. With the order of 97, Kenji ran short by 4, a cost of 112 dollars. One fortnight can be lucky or unlucky; the expected cost is the average over many.

Explore

The order quantile and its expected cost

255075100125150175order 93left over 8.4short 7.8Demand for the 14 days (units)05001,0001,500best 58%you: 50%, US$3880%10%20%30%40%50%60%70%80%90%100%Expected cost of the order (dollars)Quantile Kenji orders at
50100150order 93left over 8.4short 7.8Demand for the 14 days (units)05001,0001,500best 58%you: 50%, US$3880%25%50%75%100%Expected cost of the order (dollars)Quantile Kenji orders at

At the 50% quantile, Kenji orders 93 parasols. The chance that demand is higher than the order is 49%. The expected cost is US$388: US$220 of lost sales and US$169 of left-over items. With these costs, the best quantile is 35% ÷ (35% + 25%) = 58%: an order of 97, expected cost US$382.

At the 50% quantile
Order (units)93
Expected units short7.8
Expected units left over8.4
Expected cost of lost salesUS$220
Expected cost of left-over itemsUS$169
Expected costUS$388
Stock-out chance49%
What happened: 101 sold8 short, US$224
The parasol, 14 days from the cut-off of 8 June 2025 (ETS: mean 92.4, standard deviation 20.4, measured). Costs: a lost sale 35% of the price, a left-over item 25% (assumptions). Move the quantile and watch the expected cost.
Text description of the explorer

The explorer has one slider: the order quantile, from 1% to 99%, starting at 50%. The top chart is a bell curve of the 14-day demand, centred on 92.4, with a line at the order. To its right, orange shows the units short; to its left, green shows the units left over. The bottom chart is a U-shaped line of the expected cost. It is about 1,330 dollars at 1%, lowest at 382 dollars at the 58% quantile (a dashed green line), and about 955 dollars at 99%. A table gives the values of the chosen quantile, as in the table above, and the result with the real sales of 101.

Move the slider from 50% to 70%: the cost hardly changes. Move it to 90% and 99%: the cost rises fast. Being a little off the best quantile costs little; being far off costs a lot.

Now change the costs. Suppose a customer who finds no parasol buys elsewhere and does not come back, so a lost sale costs more than its margin. The second explorer has two more sliders.

Explore

When the costs change

255075100125150175order 97left over 10.6short 6Demand for the 14 days (units)05001,0001,500best 58%you: 58%, US$3820%10%20%30%40%50%60%70%80%90%100%Expected cost of the order (dollars)Quantile Kenji orders at
50100150order 97left over 10.6short 6Demand for the 14 days (units)05001,0001,500best 58%you: 58%, US$3820%25%50%75%100%Expected cost of the order (dollars)Quantile Kenji orders at

At the 58% quantile, Kenji orders 97 parasols. The chance that demand is higher than the order is 41%. The expected cost is US$382: US$169 of lost sales and US$213 of left-over items. With these costs, the best quantile is 35% ÷ (35% + 25%) = 58%: an order of 97, expected cost US$382.

At the 58% quantile
Order (units)97
Expected units short6
Expected units left over10.6
Expected cost of lost salesUS$169
Expected cost of left-over itemsUS$213
Expected costUS$382
Stock-out chance41%
What happened: 101 sold4 short, US$112
The same parasol range. Move the cost of a lost sale and of a left-over item (shares of the price) and watch the best quantile move.
Text description of the explorer

The same two charts, with three sliders. One is the quantile (starting at 58%), and two are the costs of a lost sale and of a left-over item (5% to 80% of the price). A dashed green line marks the critical ratio. With a lost sale at 60% and a left-over item at 25%, the best quantile moves up to 71% (0.60 ÷ 0.85): an order of 104 parasols. With a lost sale at 10%, it moves down to 29%: an order of 82. When both costs are equal, it is 50%: order the middle.

The forecast gives the range, and the business gives the quantile. When the costs change, Kenji reads a different quantile from the same range.

Larkfield: every product and every cut-off

Section titled “Larkfield: every product and every cut-off”

Priya now runs the rule over the course backtest: 650 orders. The spread of each 14-day total comes from the daily ranges, with the days taken as independent (as in A forecast is a range). She compares two rules against the real sales: order the forecast added up, or the 58% quantile.

for rule, q in [("forecast added up", None), ("the 58% quantile", ratio)]:
g = orders_at(fc_ets, q)
print(f"{rule:18} cost per order cycle: {cost_per_cycle(g)} dollars, "
f"orders too small: {(g['sold'] > g['order']).mean():.1%}")
forecast added up cost per order cycle: 16578 dollars, orders too small: 50.0%
the 58% quantile cost per order cycle: 16294 dollars, orders too small: 44.0%

An order cycle is one cut-off: one 14-day order for all 50 products. The same ETS forecast, read at the right quantile, saves 284 dollars per cycle and runs short less often. For every method that has a range (measured):

Method Cost, forecast added up Cost, the 58% quantile Stock-out chance (added up → 58%)
Seasonal naive 16,739 16,496 49.5% → 41.2%
ETS 16,578 16,294 50.0% → 44.0%
AutoARIMA 16,257 16,201 46.0% → 38.3%
Chronos-Bolt 14,750 14,348 56.6% → 53.1%
ETS with conformal intervals 16,578 16,424 50.0% → 46.5%
What the 58% quantile saves per order cycle, compared with ordering the forecast added up (measured; costs are assumptions). Every method saves. Chronos-Bolt saves the most, the conformal range less than ETS's own range.
Text description of the chart

A column chart of the saving per order cycle, from 0 to 500 dollars. Seasonal naive saves 243 dollars, ETS 284, AutoARIMA 56, Chronos 402 and the conformal range 154. Every column is above 0.

  • The quantile rule helps every method, by 0.3% to 3%, because Larkfield’s two costs are close (35% and 25%). A better range saves more. On the same point forecast, ETS’s own range saves 284 dollars a cycle, and the too-narrow conformal range only 154.
  • Chronos-Bolt has the cheapest orders of these five. But at its 58% quantile it still runs short on 53.1% of the orders. It is biased low (−5.6%), and its range is too narrow.
  • Ordering “to be safe” is not safe for the budget. ETS orders at the 90% quantile cost 18,009 dollars per cycle (your turn below).

For one seasonal product, the method matters more. Take the six cut-offs from April to August 2025, with orders at the 58% quantile (measured). The parasol’s orders cost 4,458 dollars with Chronos-Bolt and 4,806 with seasonal naive, but 7,212 with AutoARIMA and 8,004 with ETS. ETS has no yearly season: in July 2025 it ordered 456 parasols for a fortnight that sold 187.

Compare methods by the cost of their decisions

Section titled “Compare methods by the cost of their decisions”

The course’s backtest table has a cost column for this reason. It is the cost per order cycle when Kenji orders each method’s forecast added up (measured):

The cost column of the course's backtest table: the cost per order cycle of each method's orders (the forecast added up; measured, costs are assumptions). LightGBM (direct) has the lowest cost.
Text description of the chart

One row per method, most expensive first, in dollars per order cycle: drift 28,570 dollars, naive 28,261, moving average 19,444, Theta 16,812, seasonal naive 16,739, ETS 16,578, AutoARIMA 16,257, linear regression (one per product) 15,137, Chronos-Bolt 14,750, and LightGBM (direct) 13,711, the lowest, in green.

LightGBM (direct) has the lowest decision cost, 13,711 dollars, although Chronos-Bolt had the best error (WAPE 38.7% against 40.7%). Chronos-Bolt is biased low, so its errors fall on the expensive side. And the lowest error is not the lowest cost. The 28-day moving average has a better WAPE than seasonal naive (43.5% against 51.3%), but it costs more (19,444 against 16,739 dollars). Its flat line misses the weekly peaks.

Grace, the support lead, has the same problem hour by hour, from 07:00 to 21:59. One agent answers about 6 tickets an hour. The course’s costs:

  • An extra agent hour costs 28 dollars.
  • An hour one agent short costs 72 dollars: 6 tickets that wait about a day longer, at about 12 dollars each. This is an assumption made for this course: waiting has no price on any invoice. With your own data, measure it or agree on it with the business.

The critical ratio is 72 ÷ (72 + 28) = 0.72. So Grace staffs every hour to the 72% quantile of the ticket forecast, divided by 6 and rounded up.

The chart shows one day of the course backtest: Monday 9 June 2025, forecast with MSTL from the cut-off at 23:00 the evening before.

  • Real tickets ÷ 6
  • Agents at the point forecast
  • Agents at the 72% quantile
One day of roster from MSTL's forecast (measured). The 72% roster has 88 agent hours in the day, the point roster 85. The real tickets needed 74.
Text description of the chart

The chart covers the open hours from 7:00 to 21:00. Dots show the agents needed (real tickets ÷ 6): about 2 at 7:00, 7.7 at 10:00, then 3 to 6.5. A dashed blue line shows the agents at the point forecast, and a green line those at the 72% quantile. They peak at 9 and 10 at 10:00, and the green line is one agent higher at 10:00, 14:00 and 21:00. Both rosters have more agents than needed in most hours.

Over the 13 weeks of the backtest, the cost per week of each roster (measured):

Method Staff to the point forecast Staff to the 72% quantile Staff to the 90% quantile
Same hour last week 4,888 dollars 4,157 dollars 5,188 dollars
MSTL 4,044 dollars 3,718 dollars 4,876 dollars
Theta 3,883 dollars 5,845 dollars 15,042 dollars

MSTL at its 72% quantile gives the cheapest roster: 3,718 dollars a week, 326 less than at its point forecast. A short hour costs more than two extra hours, so a few more agents pay for themselves.

Many help desks set a reply-time target instead: “an agent is free for every ticket within the hour, in 9 hours out of 10”. That target is the 90% quantile. For MSTL, it costs 4,876 dollars a week, 1,158 more than the 72% roster: now Grace can put a price on the target.

When to use the newsvendor rule, when not, and what it costs

Section titled “When to use the newsvendor rule, when not, and what it costs”

Use it for a decision made before demand is known, when too little and too much both cost money: stock orders, rosters, server capacity. It costs one quantile of a range that you already have. It needs three conditions:

  • The range must be honest. If the range is too narrow or too wide at that quantile, the order is wrong (the failure case below).
  • The costs must be right. Larkfield’s costs are assumptions. If a lost sale also loses the customer, its cost is higher, and so is the quantile.
  • One decision for one period. A parasol left over in June sells in July. The 25% cost is the end-of-season discount, so the rule fits a season’s last orders best; earlier, the best quantile is higher.

Guided practice: the order and the roster in the notebook

Section titled “Guided practice: the order and the roster in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The data cell ends with the Ready: line.
  3. In section 1, predict the cheapest of the orders 93, 97 and 119. Run the cell. You see 382 dollars for the order of 97.
  4. Run the chart cell. You see the U-shaped curve and best order: 97 expected cost: 382.
  5. In the Your turn cell, compute ratio_60 (below). Run the check cell: Check passed.
  6. In section 2, run the backtest cell (about a minute). You see 16578 and 16294 dollars.
  7. In the next Your turn cell, compute cost_90. Run the check cell: Check passed.
  8. In section 3, run the roster cell. You see 4044 and 3718 dollars a week.
  9. In section 4, run the Theta cell. You see a coverage of 0.996 and 5845 dollars a week.

Change one thing: a lost sale now costs 60% of the price instead of 35%, because the customer stays away. Before you compute it, predict: does the best quantile move up or down? In the notebook, put the new critical ratio in ratio_60, rounded to 3 digits. Then compute the 90% order cost cost_90 in section 2.

Show the answer
ratio_60 = round(0.60 / (0.60 + LEFT), 3)
print(ratio_60, "-> order", int(np.ceil(MEAN + norm.ppf(ratio_60) * SD)))
0.706 -> order 104

The best quantile moves up, from 58% to 71%, and the order from 97 to 104. For the 90% rule in section 2:

g90 = orders_at(fc_ets, 0.9)
cost_90 = cost_per_cycle(g90)
print(cost_90, f"orders too small: {(g90['sold'] > g90['order']).mean():.1%}")
18009 orders too small: 20.3%

The 90% orders run short only 20.3% of the time, but they cost 1,715 dollars more per cycle than the 58% rule. The 90% rule would be right only if a lost sale cost about 9 times as much as a left-over item.

Common mistake

A quantile from a range that is too wide

Symptom: Theta had the best point forecast of the hourly tickets (WAPE 27.1%). Grace staffs to its 72% quantile. The roster costs 5,845 dollars a week, more than Theta’s own point roster (3,883) and MSTL’s 72% roster (3,718). At the 90% quantile, it costs 15,042 dollars.

Cause: Theta’s ranges are far too wide: its 80% range holds 99.6% of the real tickets. Its 72% quantile is far above the real tickets, so the roster has many idle agents. The rule read the right quantile from a wrong range.

Fix: score the range at the level that the decision uses (Score the range) before you take a quantile from it. Here, use MSTL, whose 80% range holds 85.1% of the real tickets, or build Theta a new range from many windows of past errors.

Consequence in AI work: the method with the best error can give the most expensive decision, every week, with no error message.

Check your understanding

A lost sale costs 30 dollars and a left-over item costs 10 dollars. Which quantile of demand should you order? Explain the rule with the 'last unit' argument.

Show the answer

The critical ratio is 30 ÷ (30 + 10) = 0.75, so order the 75% quantile. The last unit saves 30 dollars if it sells and costs 10 if it does not. So it pays while its chance of selling is above 10 ÷ 40 = 25%. You stop where demand is above the order 25% of the time: the 75% quantile.

Check your understanding

Predict and explain: Kenji moves from the 58% quantile to the 90% quantile 'to be safe'. What happens to the stock-out chance and to the cost, for the parasol and over the backtest?

Show the answer

The stock-out chance falls, and the cost rises. For the parasol, the order goes from 97 to 119, the stock-out chance from 41% to 10%, and the expected cost from 382 to 576 dollars. Over the backtest with ETS, the cost per order cycle rises from 16,294 to 18,009 dollars. The orders that run short fall from 44% to 20.3%. The extra parasols are mostly left over, at 25% of the price each.

Check your understanding

Tomás wants the method with the best error for both decisions: Chronos-Bolt for the stock, Theta for the roster. What do the measured costs say?

Show the answer

The best error is not the lowest cost. For the stock, Chronos-Bolt has the best WAPE (38.7%). But LightGBM (direct) has the lowest order cost: 13,711 dollars per cycle against 14,750. Chronos-Bolt is biased low, and lost sales cost more. For the roster, Theta has the best WAPE (27.1%), but its range is far too wide. Its 72% roster costs 5,845 dollars a week, against 3,718 for MSTL. Choose by the cost of the decision.

Recap

  • The newsvendor rule: order the quantile of demand at the critical ratio, lost ÷ (lost + left over). For Larkfield, 0.35 ÷ 0.60 = 0.583.
  • The parasol’s 58% order is 97 (expected cost 382 dollars); the 90% order is 119 (576 dollars). The cost curve is flat near its bottom.
  • Over the backtest, the 58% rule beats ordering the forecast for every method. ETS: 16,294 against 16,578 dollars per order cycle; at 90%, 18,009.
  • Grace staffs to the 72% quantile, from the US$72 of a short hour (an assumption) and the US$28 of an extra hour. MSTL’s roster is the cheapest: 3,718 dollars a week.
  • The lowest error is not the lowest cost. LightGBM (direct) has the cheapest orders (13,711 dollars). Theta’s too-wide range gives the most expensive roster (5,845).

You did all this once, in a notebook. Next, you put these steps into a small job that runs every week, from new data to the order and the roster.

Next: The forecasting job