From forecast to order and roster
- Lesson 24
- 45 min
- Module 7: Uncertainty and the decision
Where we are
We have quantile forecasts that we can trust. Now we make the decision. In this lesson, we use the newsvendor idea to choose the order quantile from the cost of a lost sale and a left-over item, staff the help desk the same way, and compare the cost of each method.
Lesson 24 of 27 · Forecasting and Time Series: Predict What Comes NextFrom forecast to order and roster
1. Forecasts serve decisions
2. Read a time series
3. Baselines and honest testing
4. Statistical forecasters
5. Machine learning forecasters
6. Hard series and many series
7. Uncertainty and the decision
8. Run a forecast in production
What you will be able to do
Turn a forecast range into the order and the roster with the lowest expected cost, and compare forecasting methods by the cost of their decisions.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It covers the parasol’s best order, two ordering rules over the backtest, and the roster. Its setup cells install the packages and download the Larkfield data, so it needs no local setup. About 45 minutes, with about 1 minute of waiting.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Ranges, quantiles and coverage, from A forecast is a range.
- Calibrated and sharp ranges, from Score the range.
- The hourly ticket forecasts with MSTL and Theta, from Several seasons and long series.
- The costs of the course, from Frame the forecast.
What you need on your computer
Nothing: the notebook runs in Colab or Kaggle. To run it in VS Code instead, use the folder and the environment from Meet the Larkfield data.
Tested with Python 3.12, pandas 2.3.3, statsforecast 2.1.1 and SciPy on macOS. The Larkfield data is synthetic: we made it for this course.
The newspaper seller
Section titled “The newspaper seller”A boy sells newspapers at a street corner. He buys them for 1 dollar each in the morning and sells them for 3 dollars. In the evening, today’s paper is worth nothing. How many does he buy?
If he buys too few, every customer after he runs out is a lost profit of 2 dollars. Every extra paper is a loss of 1 dollar. So he buys a little more than he expects to sell, but not a lot more.
This is the newsvendor problem, one of the oldest problems in planning. Kenji has the same one every week.
The problem: the forecast ignores the costs
Section titled “The problem: the forecast ignores the costs”When Kenji ordered the forecast added up, half of the orders ran short. The middle of the range treats a lost sale and a left-over item as if they cost the same. At Larkfield, they do not (the course’s assumptions):
- A lost sale costs the margin: 35% of the price. For a parasol at 79.99 dollars, that is 28.00 dollars.
- A left-over item is sold at a 60% discount at the end of the season. Larkfield bought it at 65% of the price and sells it at 40%, so it loses 25% of the price: 20.00 dollars for a parasol.
So Kenji should order above the middle. How far above? “The 90% quantile, to be safe” sounds careful, but every unsold extra parasol costs 20 dollars.
The idea: order up to the critical ratio
Section titled “The idea: order up to the critical ratio”Think about the last parasol of the order. Should Kenji add one more?
- If demand reaches that parasol, it sells, and it saves a lost sale: +28 dollars.
- If demand does not reach it, it is left over: −20 dollars.
Say the chance that demand reaches it is p. On average, the extra parasol is worth p × 28 − (1 − p) × 20 dollars. That is positive while p is larger than 20 ÷ 48 = 0.417. So Kenji stops where demand is above the order 41.7% of the time, and below it 58.3% of the time. That is the 58.3% quantile of demand.
The general rule is the critical ratio:
best quantile = cost of a lost sale ÷ (cost of a lost sale + cost of a left-over item)
For Larkfield: 0.35 ÷ (0.35 + 0.25) = 0.583. The price cancels out, so the same quantile works for every product. The newspaper seller buys at 2 ÷ (2 + 1) = 0.67.
A tiny example: three orders for the parasol
Section titled “A tiny example: three orders for the parasol”On Sunday 8 June 2025, ETS forecast 92.4 parasols for the next 14 days, with a standard deviation of 20.4 for the total. For a bell-shaped range like this, a formula gives the expected units short and left over for any order. The course’s forecasting job (Module 8) uses it too. Three orders:
| Order | Quantile | Expected short | Expected left over | Expected cost | Stock-out chance |
|---|---|---|---|---|---|
| 93 | 50% (the middle) | 7.8 | 8.4 | 388 dollars | 49% |
| 97 | 58% | 6.0 | 10.6 | 382 dollars | 41% |
| 119 | 90% (“to be safe”) | 0.9 | 27.5 | 576 dollars | 10% |
The expected cost is the units short × 28 dollars plus the units left over × 20 dollars. The 58% order is the cheapest, but only by 6 dollars: the bottom of the cost curve is flat. Ordering “to be safe” costs 194 dollars more. It saves about 5 lost sales (6.0 → 0.9) and adds about 17 left-over parasols (10.6 → 27.5).
Larkfield really sold 101 parasols. With the order of 97, Kenji ran short by 4, a cost of 112 dollars. One fortnight can be lucky or unlucky; the expected cost is the average over many.
Explore the newsvendor curve
Section titled “Explore the newsvendor curve”Explore
The order quantile and its expected cost
Predict first: The middle of the range is the 50% quantile. Where is the lowest expected cost: at 50%, at 58% or at 90%? How much more does the 90% order cost?
At the 50% quantile, Kenji orders 93 parasols. The chance that demand is higher than the order is 49%. The expected cost is US$388: US$220 of lost sales and US$169 of left-over items. With these costs, the best quantile is 35% ÷ (35% + 25%) = 58%: an order of 97, expected cost US$382.
| At the 50% quantile | |
|---|---|
| Order (units) | 93 |
| Expected units short | 7.8 |
| Expected units left over | 8.4 |
| Expected cost of lost sales | US$220 |
| Expected cost of left-over items | US$169 |
| Expected cost | US$388 |
| Stock-out chance | 49% |
| What happened: 101 sold | 8 short, US$224 |
Text description of the explorer
The explorer has one slider: the order quantile, from 1% to 99%, starting at 50%. The top chart is a bell curve of the 14-day demand, centred on 92.4, with a line at the order. To its right, orange shows the units short; to its left, green shows the units left over. The bottom chart is a U-shaped line of the expected cost. It is about 1,330 dollars at 1%, lowest at 382 dollars at the 58% quantile (a dashed green line), and about 955 dollars at 99%. A table gives the values of the chosen quantile, as in the table above, and the result with the real sales of 101.
Move the slider from 50% to 70%: the cost hardly changes. Move it to 90% and 99%: the cost rises fast. Being a little off the best quantile costs little; being far off costs a lot.
Now change the costs. Suppose a customer who finds no parasol buys elsewhere and does not come back, so a lost sale costs more than its margin. The second explorer has two more sliders.
Explore
When the costs change
Predict first: A lost sale now costs 60% of the price, and a left-over item still 25%. Will the best quantile move up or down? To about where?
At the 58% quantile, Kenji orders 97 parasols. The chance that demand is higher than the order is 41%. The expected cost is US$382: US$169 of lost sales and US$213 of left-over items. With these costs, the best quantile is 35% ÷ (35% + 25%) = 58%: an order of 97, expected cost US$382.
| At the 58% quantile | |
|---|---|
| Order (units) | 97 |
| Expected units short | 6 |
| Expected units left over | 10.6 |
| Expected cost of lost sales | US$169 |
| Expected cost of left-over items | US$213 |
| Expected cost | US$382 |
| Stock-out chance | 41% |
| What happened: 101 sold | 4 short, US$112 |
Text description of the explorer
The same two charts, with three sliders. One is the quantile (starting at 58%), and two are the costs of a lost sale and of a left-over item (5% to 80% of the price). A dashed green line marks the critical ratio. With a lost sale at 60% and a left-over item at 25%, the best quantile moves up to 71% (0.60 ÷ 0.85): an order of 104 parasols. With a lost sale at 10%, it moves down to 29%: an order of 82. When both costs are equal, it is 50%: order the middle.
The forecast gives the range, and the business gives the quantile. When the costs change, Kenji reads a different quantile from the same range.
Larkfield: every product and every cut-off
Section titled “Larkfield: every product and every cut-off”Priya now runs the rule over the course backtest: 650 orders. The spread of each 14-day total comes from the daily ranges, with the days taken as independent (as in A forecast is a range). She compares two rules against the real sales: order the forecast added up, or the 58% quantile.
for rule, q in [("forecast added up", None), ("the 58% quantile", ratio)]: g = orders_at(fc_ets, q) print(f"{rule:18} cost per order cycle: {cost_per_cycle(g)} dollars, " f"orders too small: {(g['sold'] > g['order']).mean():.1%}")forecast added up cost per order cycle: 16578 dollars, orders too small: 50.0%the 58% quantile cost per order cycle: 16294 dollars, orders too small: 44.0%An order cycle is one cut-off: one 14-day order for all 50 products. The same ETS forecast, read at the right quantile, saves 284 dollars per cycle and runs short less often. For every method that has a range (measured):
| Method | Cost, forecast added up | Cost, the 58% quantile | Stock-out chance (added up → 58%) |
|---|---|---|---|
| Seasonal naive | 16,739 | 16,496 | 49.5% → 41.2% |
| ETS | 16,578 | 16,294 | 50.0% → 44.0% |
| AutoARIMA | 16,257 | 16,201 | 46.0% → 38.3% |
| Chronos-Bolt | 14,750 | 14,348 | 56.6% → 53.1% |
| ETS with conformal intervals | 16,578 | 16,424 | 50.0% → 46.5% |
Text description of the chart
A column chart of the saving per order cycle, from 0 to 500 dollars. Seasonal naive saves 243 dollars, ETS 284, AutoARIMA 56, Chronos 402 and the conformal range 154. Every column is above 0.
- The quantile rule helps every method, by 0.3% to 3%, because Larkfield’s two costs are close (35% and 25%). A better range saves more. On the same point forecast, ETS’s own range saves 284 dollars a cycle, and the too-narrow conformal range only 154.
- Chronos-Bolt has the cheapest orders of these five. But at its 58% quantile it still runs short on 53.1% of the orders. It is biased low (−5.6%), and its range is too narrow.
- Ordering “to be safe” is not safe for the budget. ETS orders at the 90% quantile cost 18,009 dollars per cycle (your turn below).
The parasol’s whole season
Section titled “The parasol’s whole season”For one seasonal product, the method matters more. Take the six cut-offs from April to August 2025, with orders at the 58% quantile (measured). The parasol’s orders cost 4,458 dollars with Chronos-Bolt and 4,806 with seasonal naive, but 7,212 with AutoARIMA and 8,004 with ETS. ETS has no yearly season: in July 2025 it ordered 456 parasols for a fortnight that sold 187.
Compare methods by the cost of their decisions
Section titled “Compare methods by the cost of their decisions”The course’s backtest table has a cost column for this reason. It is the cost per order cycle when Kenji orders each method’s forecast added up (measured):
Text description of the chart
One row per method, most expensive first, in dollars per order cycle: drift 28,570 dollars, naive 28,261, moving average 19,444, Theta 16,812, seasonal naive 16,739, ETS 16,578, AutoARIMA 16,257, linear regression (one per product) 15,137, Chronos-Bolt 14,750, and LightGBM (direct) 13,711, the lowest, in green.
LightGBM (direct) has the lowest decision cost, 13,711 dollars, although Chronos-Bolt had the best error (WAPE 38.7% against 40.7%). Chronos-Bolt is biased low, so its errors fall on the expensive side. And the lowest error is not the lowest cost. The 28-day moving average has a better WAPE than seasonal naive (43.5% against 51.3%), but it costs more (19,444 against 16,739 dollars). Its flat line misses the weekly peaks.
The help-desk roster
Section titled “The help-desk roster”Grace, the support lead, has the same problem hour by hour, from 07:00 to 21:59. One agent answers about 6 tickets an hour. The course’s costs:
- An extra agent hour costs 28 dollars.
- An hour one agent short costs 72 dollars: 6 tickets that wait about a day longer, at about 12 dollars each. This is an assumption made for this course: waiting has no price on any invoice. With your own data, measure it or agree on it with the business.
The critical ratio is 72 ÷ (72 + 28) = 0.72. So Grace staffs every hour to the 72% quantile of the ticket forecast, divided by 6 and rounded up.
The chart shows one day of the course backtest: Monday 9 June 2025, forecast with MSTL from the cut-off at 23:00 the evening before.
- Real tickets ÷ 6
- Agents at the point forecast
- Agents at the 72% quantile
Text description of the chart
The chart covers the open hours from 7:00 to 21:00. Dots show the agents needed (real tickets ÷ 6): about 2 at 7:00, 7.7 at 10:00, then 3 to 6.5. A dashed blue line shows the agents at the point forecast, and a green line those at the 72% quantile. They peak at 9 and 10 at 10:00, and the green line is one agent higher at 10:00, 14:00 and 21:00. Both rosters have more agents than needed in most hours.
Over the 13 weeks of the backtest, the cost per week of each roster (measured):
| Method | Staff to the point forecast | Staff to the 72% quantile | Staff to the 90% quantile |
|---|---|---|---|
| Same hour last week | 4,888 dollars | 4,157 dollars | 5,188 dollars |
| MSTL | 4,044 dollars | 3,718 dollars | 4,876 dollars |
| Theta | 3,883 dollars | 5,845 dollars | 15,042 dollars |
MSTL at its 72% quantile gives the cheapest roster: 3,718 dollars a week, 326 less than at its point forecast. A short hour costs more than two extra hours, so a few more agents pay for themselves.
Many help desks set a reply-time target instead: “an agent is free for every ticket within the hour, in 9 hours out of 10”. That target is the 90% quantile. For MSTL, it costs 4,876 dollars a week, 1,158 more than the 72% roster: now Grace can put a price on the target.
When to use the newsvendor rule, when not, and what it costs
Section titled “When to use the newsvendor rule, when not, and what it costs”Use it for a decision made before demand is known, when too little and too much both cost money: stock orders, rosters, server capacity. It costs one quantile of a range that you already have. It needs three conditions:
- The range must be honest. If the range is too narrow or too wide at that quantile, the order is wrong (the failure case below).
- The costs must be right. Larkfield’s costs are assumptions. If a lost sale also loses the customer, its cost is higher, and so is the quantile.
- One decision for one period. A parasol left over in June sells in July. The 25% cost is the end-of-season discount, so the rule fits a season’s last orders best; earlier, the best quantile is higher.
Guided practice: the order and the roster in the notebook
Section titled “Guided practice: the order and the roster in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with the
Ready:line. - In section 1, predict the cheapest of the orders 93, 97 and 119. Run the cell. You see
382 dollarsfor the order of 97. - Run the chart cell. You see the U-shaped curve and
best order: 97 expected cost: 382. - In the Your turn cell, compute
ratio_60(below). Run the check cell:Check passed. - In section 2, run the backtest cell (about a minute). You see
16578and16294dollars. - In the next Your turn cell, compute
cost_90. Run the check cell:Check passed. - In section 3, run the roster cell. You see
4044and3718dollars a week. - In section 4, run the Theta cell. You see a coverage of
0.996and5845dollars a week.
Your turn: a lost customer
Section titled “Your turn: a lost customer”Change one thing: a lost sale now costs 60% of the price instead of 35%, because the customer stays away. Before you compute it, predict: does the best quantile move up or down? In the notebook, put the new critical ratio in ratio_60, rounded to 3 digits. Then compute the 90% order cost cost_90 in section 2.
Show the answer
ratio_60 = round(0.60 / (0.60 + LEFT), 3)print(ratio_60, "-> order", int(np.ceil(MEAN + norm.ppf(ratio_60) * SD)))0.706 -> order 104The best quantile moves up, from 58% to 71%, and the order from 97 to 104. For the 90% rule in section 2:
g90 = orders_at(fc_ets, 0.9)cost_90 = cost_per_cycle(g90)print(cost_90, f"orders too small: {(g90['sold'] > g90['order']).mean():.1%}")18009 orders too small: 20.3%The 90% orders run short only 20.3% of the time, but they cost 1,715 dollars more per cycle than the 58% rule. The 90% rule would be right only if a lost sale cost about 9 times as much as a left-over item.
Common mistake
A quantile from a range that is too wide
Symptom: Theta had the best point forecast of the hourly tickets (WAPE 27.1%). Grace staffs to its 72% quantile. The roster costs 5,845 dollars a week, more than Theta’s own point roster (3,883) and MSTL’s 72% roster (3,718). At the 90% quantile, it costs 15,042 dollars.
Cause: Theta’s ranges are far too wide: its 80% range holds 99.6% of the real tickets. Its 72% quantile is far above the real tickets, so the roster has many idle agents. The rule read the right quantile from a wrong range.
Fix: score the range at the level that the decision uses (Score the range) before you take a quantile from it. Here, use MSTL, whose 80% range holds 85.1% of the real tickets, or build Theta a new range from many windows of past errors.
Consequence in AI work: the method with the best error can give the most expensive decision, every week, with no error message.
Knowledge check
Section titled “Knowledge check”Check your understanding
A lost sale costs 30 dollars and a left-over item costs 10 dollars. Which quantile of demand should you order? Explain the rule with the 'last unit' argument.
Show the answer
The critical ratio is 30 ÷ (30 + 10) = 0.75, so order the 75% quantile. The last unit saves 30 dollars if it sells and costs 10 if it does not. So it pays while its chance of selling is above 10 ÷ 40 = 25%. You stop where demand is above the order 25% of the time: the 75% quantile.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict and explain: Kenji moves from the 58% quantile to the 90% quantile 'to be safe'. What happens to the stock-out chance and to the cost, for the parasol and over the backtest?
Show the answer
The stock-out chance falls, and the cost rises. For the parasol, the order goes from 97 to 119, the stock-out chance from 41% to 10%, and the expected cost from 382 to 576 dollars. Over the backtest with ETS, the cost per order cycle rises from 16,294 to 18,009 dollars. The orders that run short fall from 44% to 20.3%. The extra parasols are mostly left over, at 25% of the price each.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás wants the method with the best error for both decisions: Chronos-Bolt for the stock, Theta for the roster. What do the measured costs say?
Show the answer
The best error is not the lowest cost. For the stock, Chronos-Bolt has the best WAPE (38.7%). But LightGBM (direct) has the lowest order cost: 13,711 dollars per cycle against 14,750. Chronos-Bolt is biased low, and lost sales cost more. For the roster, Theta has the best WAPE (27.1%), but its range is far too wide. Its 72% roster costs 5,845 dollars a week, against 3,718 for MSTL. Choose by the cost of the decision.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- The newsvendor rule: order the quantile of demand at the critical ratio, lost ÷ (lost + left over). For Larkfield, 0.35 ÷ 0.60 = 0.583.
- The parasol’s 58% order is 97 (expected cost 382 dollars); the 90% order is 119 (576 dollars). The cost curve is flat near its bottom.
- Over the backtest, the 58% rule beats ordering the forecast for every method. ETS: 16,294 against 16,578 dollars per order cycle; at 90%, 18,009.
- Grace staffs to the 72% quantile, from the US$72 of a short hour (an assumption) and the US$28 of an extra hour. MSTL’s roster is the cheapest: 3,718 dollars a week.
- The lowest error is not the lowest cost. LightGBM (direct) has the cheapest orders (13,711 dollars). Theta’s too-wide range gives the most expensive roster (5,845).
You did all this once, in a notebook. Next, you put these steps into a small job that runs every week, from new data to the order and the roster.
Next: The forecasting job