Skip to content
Français

Score the range

  • Lesson 23
  • 35 min
  • Module 7: Uncertainty and the decision

Where we are

We now have forecast ranges. A range can be too narrow or too wide, and both cost money. In this lesson, we score a range with coverage and pinball loss, and we fix a range that is too narrow with conformal intervals.

What you will be able to do

Score a forecast range so that you can compare the ranges of two methods and see which one to trust for a decision. A range that is too narrow promises safety that it does not give; a range that is too wide makes Kenji order too much.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson, with the instructions. It computes the pinball loss of one day and compares two methods’ ranges over the course backtest. Then it tests conformal intervals. Its setup cells install the packages and download the Larkfield data, so it needs no local setup. About 35 minutes, with about 2 minutes of waiting.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

Nothing: the notebook runs in Colab or Kaggle. To run it in VS Code instead, use the folder and the environment from Meet the Larkfield data.

Tested with Python 3.12, pandas 2.3.3 and statsforecast 2.1.1 on macOS. The Larkfield data is synthetic: we made it for this course.

Imagine two weather forecasters in your town. The first one says every evening: “Tomorrow, the temperature will be between −20 and +45 degrees.” She is never wrong. But her forecast is useless: you cannot choose a coat from it.

The second one says every evening: “Tomorrow, exactly 21 degrees.” His forecast is precise. But it is almost never right, and he does not tell you how far off he can be.

You want a third forecaster. She says: “Between 18 and 24 degrees, and I am right about 8 days in 10.” And when you check her for a year, she really is right about 8 days in 10. Her range is honest, and it is narrow enough to use.

The problem: coverage alone is easy to win

Section titled “The problem: coverage alone is easy to win”

In the last lesson, we checked a range with its coverage: the share of real values inside it. Coverage catches the second forecaster. A range that is too narrow holds too few real values.

But coverage cannot catch the first forecaster. Look at a range from 0 to 1,000 parasols a day. It holds every real value: its coverage is 100%. It promised 80%, so it is “too safe”, but nobody would call it a better range than one that holds 83%. Coverage only says whether the promise is kept. It does not reward a range for being narrow.

So we need two things. A range must be calibrated: its coverage is close to its promise. And it must be sharp: as narrow as it can be while it keeps the promise. The pinball loss scores both at once.

Here is the problem on Larkfield’s data. In the notebook, Priya compares two methods over the course backtest: the seasonal naive forecast and ETS. Both give 80% and 90% ranges.

Method Coverage of the 80% range Coverage of the 90% range Mean width of the 80% range
Seasonal naive 87.7% 92.9% 13.5 units
ETS 83.5% 90.5% 9.6 units

If we look only at coverage, the seasonal naive range looks “safer”: 87.7% instead of 83.5%. But it is 40% wider. Both keep their promise. Which one is better for Kenji? The narrower one, as long as it keeps its promise, because every unit of width makes him order more stock.

The pinball loss scores one quantile forecast at a time, with a simple rule. Take a q90 forecast. It says: “the real value is below me 9 times in 10”. So it should almost never be too low.

  • If the real value is above the q90, each unit of the gap costs 0.9.
  • If the real value is below the q90, each unit of the gap costs only 0.1.

Write the level of a quantile as τ, the Greek letter “tau”: a number between 0 and 1. A gap above costs τ per unit, and a gap below costs 1 − τ per unit. The q90 has τ = 0.9. The median has τ = 0.5, so both sides cost 0.5: for the median, the pinball loss is half the absolute error. The name comes from the shape of its graph: a V that leans to one side.

Why this rule? Because the cheapest place for a q90, over many days, is exactly where the real value is below it 9 times in 10. Put it higher, and you pay 0.1 on almost every day. Put it lower, and you pay 0.9 on more days. Only the honest q90 balances the two. So a low pinball loss needs each quantile in the right place, and a range that is too wide pays for its width.

The q90 forecast is 26 units. Two real days:

Real value Gap Side Cost per unit Pinball loss
20 26 − 20 = 6 below the q90 0.1 6 × 0.1 = 0.6
30 30 − 26 = 4 above the q90 0.9 4 × 0.9 = 3.6

The day of 30 costs 6 times more, with a smaller gap. The chart shows the loss of this q90, and of a median of 26, for every real value from 10 to 40.

  • q90 forecast of 26
  • median (q50) forecast of 26
The pinball loss of a forecast of 26 units (constructed example). For the q90 (solid), a real value above 26 costs 0.9 per unit and a value below costs 0.1 per unit. For the median (dashed), both sides cost 0.5 per unit.
Text description of the chart

The horizontal axis is the real value, from 10 to 40 units. The vertical axis is the pinball loss. Both lines are 0 at 26, the forecast. The solid line, for the q90, rises slowly to the left: at 10 it is only 1.6. To the right it rises steeply: at 40 it is 12.6. Two points are marked: 0.6 at a real value of 20, and 3.6 at 30. The dashed line, for the median, is a symmetric V: 8 at 10 and 7 at 40.

In the backtest, bt.score computes the pinball loss of every quantile column (q5, q10, q50, q90, q95) on every forecast day, and takes the mean. Then it divides by the mean real value. So a product that sells 100 a day does not count more than a product that sells 3. You saw this column in the course’s backtest table, in Measure the error that matters. Lower is better.

Priya scores the ranges of every method that gives one, over the course backtest (measured, 9,100 forecast days each):

Method Coverage 80% Coverage 90% Pinball loss
Seasonal naive 87.7% 92.9% 0.133
ETS 83.5% 90.5% 0.110
AutoARIMA 83.0% 90.2% 0.110
Chronos-Bolt 78.9% 78.9% 0.099
ETS with conformal intervals 68.9% 75.6% 0.116
The coverage of each method's 90% range over the course backtest (measured). The promise is 90%. Blue: the promise is kept. Orange: the range is too narrow. Chronos-Bolt has no real 90% level, and the conformal range holds only 75.6%.
Text description of the chart

A column chart with five columns, on an axis from 0 to 100%. Seasonal naive (S. naive) 92.9%, ETS 90.5% and AutoARIMA (ARIMA) 90.2% are blue: they keep the promise of 90%. Chronos 78.9% and Conformal 75.6% are orange: they hold far fewer real values than they promise.

What this table tells us, row by row:

  • Seasonal naive keeps its promise with room to spare, but its pinball loss is the worst, 0.133. Its range is wide, and it pays for that width.
  • ETS and AutoARIMA keep both promises closely, with the same pinball loss of 0.110. AutoARIMA took about 21 times longer to fit, for no better range.
  • Chronos-Bolt has the lowest pinball loss, 0.099, because its quantiles are sharp: they sit close to the real values. But its 80% range holds only 78.9%, a little too narrow. And it predicts only the quantiles from 0.1 to 0.9, so its “90% range” is its 80% range: it holds 78.9% too. For a decision that needs a q95, Chronos-Bolt cannot give one.
  • ETS with conformal intervals, a method that should fix ranges that are too narrow, made them much narrower. The failure case below explains why.

Tomás reads the pinball column and says: “Chronos wins again.” Priya agrees that its quantiles are the sharpest. Then she points at the coverage: “One score cannot show everything. The pinball loss is an average over five quantiles. It rewards sharp quantiles, and it hides that the q5 and the q95 are missing. Look at coverage and pinball together.”

What if a model’s range is too narrow, or the model gives no range at all, like LightGBM? Conformal intervals build a range from the model’s own past errors. Think of the bus again. You do not need a theory of traffic. You write down how late the bus was on the last 20 trips. Suppose it was at most 12 minutes late on 18 of the 20 trips. Then you leave 12 minutes early, and you are on time about 9 times in 10.

statsforecast does this with windows. Before the cut-off, it hides a stretch of history, forecasts it, and compares the forecast with the real values. Each stretch is one window. With 4 windows of 14 days, it gets 4 errors for day 1 ahead, 4 errors for day 2, and so on. For the new forecast, it puts the range around the point forecast from the size of those errors.

A flow from left to right. A box of history before the cut-off holds four windows of 14 days, each a forecast followed by the real values. An arrow leads to Errors: 4 per day ahead. Then Size of the range from the errors: a large one for 90%. Then New forecast after the cut-off, plus or minus that size.
Conformal intervals: forecast a few earlier windows, measure the errors, and put the new range around the new forecast from the size of those errors.
Text description of the diagram

The diagram starts with a box called History before the cut-off. Inside it are four boxes in a row: Window 1, Window 2, Window 3 and Window 4. Each window is 14 days: the model forecasts it, then the real values are compared with the forecast. An arrow leads out of the history box to Errors: 4 errors for each day ahead, day 1, day 2 and so on. The next box is Size of the range, from the errors: a large error for a 90% range. The last box, with a thick border, is New forecast after the cut-off, plus or minus that size.

The method has a strong side. It makes no assumption about the shape of the errors: it uses the errors as they are. It works around any point forecaster, also around LightGBM or a forecast from another team.

It also has a cost. Each window is one more fit of the model. With 4 windows, ETS is fitted 5 times at each cut-off. In the course backtest, ETS with its own ranges took about 46 seconds; with conformal intervals, about 211 seconds.

Guided practice: score two ranges in the notebook

Section titled “Guided practice: score two ranges in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The data cell ends with the Ready: line.
  3. In section 1, predict which day has the larger loss. Run the cell. You see loss 0.6 and loss 3.6.
  4. In section 2, predict which method has the wider ranges. Run the backtest cell. It takes about a minute.
  5. Compare the table with the one above. You see a width of 13.5 for the seasonal naive range and 9.6 for ETS.
  6. In section 3, compute above_q90, as described below. Run the check cell. It prints Check passed.
  7. In section 4, predict the coverage of the conformal range. Run the cell. It takes about a minute and a half.
  8. Read the two lines. They cover the last 4 cut-offs only. The conformal 90% range holds 0.75 of the real values; ETS’s own holds 0.909.

Coverage counts the values inside the range. It does not say on which side the others fall. For Kenji, a miss above the range is a stock-out, and a miss below is extra stock. Before you run it, predict: does ETS miss more often above its q90, or below its q10? In the notebook, compute the share of real values above the q90 column of fc_ets, and put it in above_q90, rounded to 3 digits.

Show the answer
above_q90 = round(float((fc_ets["y"] > fc_ets["q90"]).mean()), 3)
print("above q90:", above_q90, " below q10:", round(float((fc_ets["y"] < fc_ets["q10"]).mean()), 3))
above q90: 0.088 below q10: 0.077

Each side should hold 10% of the real values. ETS has 8.8% above its q90 and 7.7% below its q10. Both are a little under 10%, so the range is a little wide on both sides. It is not skewed: it does not miss much more often on the stock-out side.

Common mistake

Conformal intervals from 4 windows

Symptom: Priya adds conformal intervals to ETS, with 4 windows of 14 days. Nothing fails, and the forecast looks the same. But over the course backtest, the new 80% range holds only 68.9% of the real values, and the 90% range only 75.6%. ETS’s own ranges held 83.5% and 90.5%. In the notebook, on the last 4 cut-offs, the conformal range holds 68.4% and 75.0%, against 84.3% and 90.9% for ETS’s own range. The “fix” made the range much too narrow.

Cause: for each day ahead, the method has only 4 past errors. A range from 4 errors cannot hold 90%. Think of 5 errors in a row: the 4 old ones and the new one. If the errors are of the same kind, each of the 5 is equally likely to be the largest. So the new error is larger than all 4 old ones about 1 time in 5. A range as wide as the largest old error holds at most about 80%. The 4 windows also come from the last 8 weeks only. A calm stretch before the cut-off then gives a narrow range for a busy period after it.

Fix: use many windows, so that the range comes from many errors. With 20 windows, a new error is larger than all of them only about 1 time in 21. Each window is one more fit, so this costs time. For a model that already has an honest range, such as ETS here, keep its own range. Use conformal intervals for a model that has no range, and check the coverage on a backtest, as here.

Consequence in AI work: a method with a good reputation can make things worse in your setting, and it raises no error. Only a measurement shows it. Score every range on a backtest before a decision uses it.

When to use each score, and what they cost

Section titled “When to use each score, and what they cost”

Use coverage first: it answers a plain question that a planner understands (“does the 90% range hold 90%?”). Check it at the level that your decision uses. Use the pinball loss to compare ranges that all keep their promise: it rewards the sharpest one. Report both, next to the point error, in the backtest table. Both are cheap: they need only the quantile columns that the backtest already keeps.

Neither score is the decision cost. A range can score well and still lead to a poor order, when the order needs a quantile that the range scores badly at. The next lesson measures the cost of the decisions themselves.

Check your understanding

A range from 0 to 1,000 units holds every real value. Why is it not a good range? Which score shows that?

Show the answer

It keeps its promise (coverage 100% for an 80% promise), but it is far too wide to plan with: Kenji would order far too much. Coverage cannot show that, because it only counts the values inside. The pinball loss shows it: its q90 of 1,000 sits far above the real values, so it pays 0.1 per unit of a very large gap on every day. Its q10 of 0 sits far below them, and it also pays 0.1 per unit of its gap on almost every day. A good range is calibrated (coverage close to the promise) and sharp (as narrow as possible).

Check your understanding

Predict and explain: a q90 forecast is 50. The real value is 40 on one day and 60 on another day. Which day has the larger pinball loss, and how many times larger?

Show the answer

The day of 60. Above the q90, each unit costs 0.9: 10 × 0.9 = 9. Below it, each unit costs 0.1: 10 × 0.1 = 1. The gaps are the same, 10 units, but the day above costs 9 times more, because a q90 should almost never be too low.

Check your understanding

Tomás says: 'Chronos-Bolt has the lowest pinball loss, so we use its 90% range for the safety stock.' What do you answer?

Show the answer

Its quantiles are the sharpest (pinball 0.099), but it predicts only the quantiles from 0.1 to 0.9. Its “90% range” is its 80% range, and it holds only 78.9% of the real values. A safety stock at its “q95” would run short about twice as often as planned. For a decision at the 90% or 95% level, use a method whose range keeps its promise at that level on the backtest. ETS does (90.5%).

Recap

  • A good range is calibrated (its coverage is close to its promise) and sharp (as narrow as it can be). Coverage checks the first. It cannot catch a range that is too wide.
  • The pinball loss scores each quantile: for a q90, a real value above costs 0.9 per unit and a value below costs 0.1. It rewards quantiles in the right place, and it makes a wide range pay for its width.
  • On Larkfield, ETS and AutoARIMA keep both promises (83.5% and 90.5%; 83.0% and 90.2%) with a pinball loss of 0.110. Seasonal naive is too wide (0.133). Chronos-Bolt is the sharpest (0.099), but it holds only 78.9% and has no real 90% level.
  • Conformal intervals build a range from past errors. With 4 windows, they made the ETS range worse: 68.9% and 75.6%. They need many windows, and each window is one more fit.

You can now tell a good range from a poor one. But Kenji still needs one number to order, and Grace one number of agents per hour. Next, you choose the quantile from the costs, and you compare the cost of each method’s decisions.

Next: From forecast to order and roster