Score the range
- Lesson 23
- 35 min
- Module 7: Uncertainty and the decision
Where we are
We now have forecast ranges. A range can be too narrow or too wide, and both cost money. In this lesson, we score a range with coverage and pinball loss, and we fix a range that is too narrow with conformal intervals.
Lesson 23 of 27 · Forecasting and Time Series: Predict What Comes NextScore the range
1. Forecasts serve decisions
2. Read a time series
3. Baselines and honest testing
4. Statistical forecasters
5. Machine learning forecasters
6. Hard series and many series
7. Uncertainty and the decision
8. Run a forecast in production
What you will be able to do
Score a forecast range so that you can compare the ranges of two methods and see which one to trust for a decision. A range that is too narrow promises safety that it does not give; a range that is too wide makes Kenji order too much.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson, with the instructions. It computes the pinball loss of one day and compares two methods’ ranges over the course backtest. Then it tests conformal intervals. Its setup cells install the packages and download the Larkfield data, so it needs no local setup. About 35 minutes, with about 2 minutes of waiting.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Prediction intervals, quantiles (q10, q90) and coverage, from A forecast is a range.
- The seasonal naive forecast (the same weekday last week), from Naive forecasts.
- The course backtest and its table, from Test like the future: backtesting.
What you need on your computer
Nothing: the notebook runs in Colab or Kaggle. To run it in VS Code instead, use the folder and the environment from Meet the Larkfield data.
Tested with Python 3.12, pandas 2.3.3 and statsforecast 2.1.1 on macOS. The Larkfield data is synthetic: we made it for this course.
Two weather forecasters
Section titled “Two weather forecasters”Imagine two weather forecasters in your town. The first one says every evening: “Tomorrow, the temperature will be between −20 and +45 degrees.” She is never wrong. But her forecast is useless: you cannot choose a coat from it.
The second one says every evening: “Tomorrow, exactly 21 degrees.” His forecast is precise. But it is almost never right, and he does not tell you how far off he can be.
You want a third forecaster. She says: “Between 18 and 24 degrees, and I am right about 8 days in 10.” And when you check her for a year, she really is right about 8 days in 10. Her range is honest, and it is narrow enough to use.
The problem: coverage alone is easy to win
Section titled “The problem: coverage alone is easy to win”In the last lesson, we checked a range with its coverage: the share of real values inside it. Coverage catches the second forecaster. A range that is too narrow holds too few real values.
But coverage cannot catch the first forecaster. Look at a range from 0 to 1,000 parasols a day. It holds every real value: its coverage is 100%. It promised 80%, so it is “too safe”, but nobody would call it a better range than one that holds 83%. Coverage only says whether the promise is kept. It does not reward a range for being narrow.
So we need two things. A range must be calibrated: its coverage is close to its promise. And it must be sharp: as narrow as it can be while it keeps the promise. The pinball loss scores both at once.
Here is the problem on Larkfield’s data. In the notebook, Priya compares two methods over the course backtest: the seasonal naive forecast and ETS. Both give 80% and 90% ranges.
| Method | Coverage of the 80% range | Coverage of the 90% range | Mean width of the 80% range |
|---|---|---|---|
| Seasonal naive | 87.7% | 92.9% | 13.5 units |
| ETS | 83.5% | 90.5% | 9.6 units |
If we look only at coverage, the seasonal naive range looks “safer”: 87.7% instead of 83.5%. But it is 40% wider. Both keep their promise. Which one is better for Kenji? The narrower one, as long as it keeps its promise, because every unit of width makes him order more stock.
The idea: a score for each quantile
Section titled “The idea: a score for each quantile”The pinball loss scores one quantile forecast at a time, with a simple rule. Take a q90 forecast. It says: “the real value is below me 9 times in 10”. So it should almost never be too low.
- If the real value is above the q90, each unit of the gap costs 0.9.
- If the real value is below the q90, each unit of the gap costs only 0.1.
Write the level of a quantile as τ, the Greek letter “tau”: a number between 0 and 1. A gap above costs τ per unit, and a gap below costs 1 − τ per unit. The q90 has τ = 0.9. The median has τ = 0.5, so both sides cost 0.5: for the median, the pinball loss is half the absolute error. The name comes from the shape of its graph: a V that leans to one side.
Why this rule? Because the cheapest place for a q90, over many days, is exactly where the real value is below it 9 times in 10. Put it higher, and you pay 0.1 on almost every day. Put it lower, and you pay 0.9 on more days. Only the honest q90 balances the two. So a low pinball loss needs each quantile in the right place, and a range that is too wide pays for its width.
A tiny example: one q90 and two days
Section titled “A tiny example: one q90 and two days”The q90 forecast is 26 units. Two real days:
| Real value | Gap | Side | Cost per unit | Pinball loss |
|---|---|---|---|---|
| 20 | 26 − 20 = 6 | below the q90 | 0.1 | 6 × 0.1 = 0.6 |
| 30 | 30 − 26 = 4 | above the q90 | 0.9 | 4 × 0.9 = 3.6 |
The day of 30 costs 6 times more, with a smaller gap. The chart shows the loss of this q90, and of a median of 26, for every real value from 10 to 40.
- q90 forecast of 26
- median (q50) forecast of 26
Text description of the chart
The horizontal axis is the real value, from 10 to 40 units. The vertical axis is the pinball loss. Both lines are 0 at 26, the forecast. The solid line, for the q90, rises slowly to the left: at 10 it is only 1.6. To the right it rises steeply: at 40 it is 12.6. Two points are marked: 0.6 at a real value of 20, and 3.6 at 30. The dashed line, for the median, is a symmetric V: 8 at 10 and 7 at 40.
In the backtest, bt.score computes the pinball loss of every quantile column (q5, q10, q50, q90, q95) on every forecast day, and takes the mean. Then it divides by the mean real value. So a product that sells 100 a day does not count more than a product that sells 3. You saw this column in the course’s backtest table, in Measure the error that matters. Lower is better.
Larkfield: the ranges of every method
Section titled “Larkfield: the ranges of every method”Priya scores the ranges of every method that gives one, over the course backtest (measured, 9,100 forecast days each):
| Method | Coverage 80% | Coverage 90% | Pinball loss |
|---|---|---|---|
| Seasonal naive | 87.7% | 92.9% | 0.133 |
| ETS | 83.5% | 90.5% | 0.110 |
| AutoARIMA | 83.0% | 90.2% | 0.110 |
| Chronos-Bolt | 78.9% | 78.9% | 0.099 |
| ETS with conformal intervals | 68.9% | 75.6% | 0.116 |
Text description of the chart
A column chart with five columns, on an axis from 0 to 100%. Seasonal naive (S. naive) 92.9%, ETS 90.5% and AutoARIMA (ARIMA) 90.2% are blue: they keep the promise of 90%. Chronos 78.9% and Conformal 75.6% are orange: they hold far fewer real values than they promise.
What this table tells us, row by row:
- Seasonal naive keeps its promise with room to spare, but its pinball loss is the worst, 0.133. Its range is wide, and it pays for that width.
- ETS and AutoARIMA keep both promises closely, with the same pinball loss of 0.110. AutoARIMA took about 21 times longer to fit, for no better range.
- Chronos-Bolt has the lowest pinball loss, 0.099, because its quantiles are sharp: they sit close to the real values. But its 80% range holds only 78.9%, a little too narrow. And it predicts only the quantiles from 0.1 to 0.9, so its “90% range” is its 80% range: it holds 78.9% too. For a decision that needs a q95, Chronos-Bolt cannot give one.
- ETS with conformal intervals, a method that should fix ranges that are too narrow, made them much narrower. The failure case below explains why.
Tomás reads the pinball column and says: “Chronos wins again.” Priya agrees that its quantiles are the sharpest. Then she points at the coverage: “One score cannot show everything. The pinball loss is an average over five quantiles. It rewards sharp quantiles, and it hides that the q5 and the q95 are missing. Look at coverage and pinball together.”
A simple fix: conformal intervals
Section titled “A simple fix: conformal intervals”What if a model’s range is too narrow, or the model gives no range at all, like LightGBM? Conformal intervals build a range from the model’s own past errors. Think of the bus again. You do not need a theory of traffic. You write down how late the bus was on the last 20 trips. Suppose it was at most 12 minutes late on 18 of the 20 trips. Then you leave 12 minutes early, and you are on time about 9 times in 10.
statsforecast does this with windows. Before the cut-off, it hides a stretch of history, forecasts it, and compares the forecast with the real values. Each stretch is one window. With 4 windows of 14 days, it gets 4 errors for day 1 ahead, 4 errors for day 2, and so on. For the new forecast, it puts the range around the point forecast from the size of those errors.
Text description of the diagram
The diagram starts with a box called History before the cut-off. Inside it are four boxes in a row: Window 1, Window 2, Window 3 and Window 4. Each window is 14 days: the model forecasts it, then the real values are compared with the forecast. An arrow leads out of the history box to Errors: 4 errors for each day ahead, day 1, day 2 and so on. The next box is Size of the range, from the errors: a large error for a 90% range. The last box, with a thick border, is New forecast after the cut-off, plus or minus that size.
The method has a strong side. It makes no assumption about the shape of the errors: it uses the errors as they are. It works around any point forecaster, also around LightGBM or a forecast from another team.
It also has a cost. Each window is one more fit of the model. With 4 windows, ETS is fitted 5 times at each cut-off. In the course backtest, ETS with its own ranges took about 46 seconds; with conformal intervals, about 211 seconds.
Guided practice: score two ranges in the notebook
Section titled “Guided practice: score two ranges in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with the
Ready:line. - In section 1, predict which day has the larger loss. Run the cell. You see
loss 0.6andloss 3.6. - In section 2, predict which method has the wider ranges. Run the backtest cell. It takes about a minute.
- Compare the table with the one above. You see a width of
13.5for the seasonal naive range and9.6for ETS. - In section 3, compute
above_q90, as described below. Run the check cell. It printsCheck passed. - In section 4, predict the coverage of the conformal range. Run the cell. It takes about a minute and a half.
- Read the two lines. They cover the last 4 cut-offs only. The conformal 90% range holds
0.75of the real values; ETS’s own holds0.909.
Your turn: where the misses fall
Section titled “Your turn: where the misses fall”Coverage counts the values inside the range. It does not say on which side the others fall. For Kenji, a miss above the range is a stock-out, and a miss below is extra stock. Before you run it, predict: does ETS miss more often above its q90, or below its q10? In the notebook, compute the share of real values above the q90 column of fc_ets, and put it in above_q90, rounded to 3 digits.
Show the answer
above_q90 = round(float((fc_ets["y"] > fc_ets["q90"]).mean()), 3)print("above q90:", above_q90, " below q10:", round(float((fc_ets["y"] < fc_ets["q10"]).mean()), 3))above q90: 0.088 below q10: 0.077Each side should hold 10% of the real values. ETS has 8.8% above its q90 and 7.7% below its q10. Both are a little under 10%, so the range is a little wide on both sides. It is not skewed: it does not miss much more often on the stock-out side.
Common mistake
Conformal intervals from 4 windows
Symptom: Priya adds conformal intervals to ETS, with 4 windows of 14 days. Nothing fails, and the forecast looks the same. But over the course backtest, the new 80% range holds only 68.9% of the real values, and the 90% range only 75.6%. ETS’s own ranges held 83.5% and 90.5%. In the notebook, on the last 4 cut-offs, the conformal range holds 68.4% and 75.0%, against 84.3% and 90.9% for ETS’s own range. The “fix” made the range much too narrow.
Cause: for each day ahead, the method has only 4 past errors. A range from 4 errors cannot hold 90%. Think of 5 errors in a row: the 4 old ones and the new one. If the errors are of the same kind, each of the 5 is equally likely to be the largest. So the new error is larger than all 4 old ones about 1 time in 5. A range as wide as the largest old error holds at most about 80%. The 4 windows also come from the last 8 weeks only. A calm stretch before the cut-off then gives a narrow range for a busy period after it.
Fix: use many windows, so that the range comes from many errors. With 20 windows, a new error is larger than all of them only about 1 time in 21. Each window is one more fit, so this costs time. For a model that already has an honest range, such as ETS here, keep its own range. Use conformal intervals for a model that has no range, and check the coverage on a backtest, as here.
Consequence in AI work: a method with a good reputation can make things worse in your setting, and it raises no error. Only a measurement shows it. Score every range on a backtest before a decision uses it.
When to use each score, and what they cost
Section titled “When to use each score, and what they cost”Use coverage first: it answers a plain question that a planner understands (“does the 90% range hold 90%?”). Check it at the level that your decision uses. Use the pinball loss to compare ranges that all keep their promise: it rewards the sharpest one. Report both, next to the point error, in the backtest table. Both are cheap: they need only the quantile columns that the backtest already keeps.
Neither score is the decision cost. A range can score well and still lead to a poor order, when the order needs a quantile that the range scores badly at. The next lesson measures the cost of the decisions themselves.
Knowledge check
Section titled “Knowledge check”Check your understanding
A range from 0 to 1,000 units holds every real value. Why is it not a good range? Which score shows that?
Show the answer
It keeps its promise (coverage 100% for an 80% promise), but it is far too wide to plan with: Kenji would order far too much. Coverage cannot show that, because it only counts the values inside. The pinball loss shows it: its q90 of 1,000 sits far above the real values, so it pays 0.1 per unit of a very large gap on every day. Its q10 of 0 sits far below them, and it also pays 0.1 per unit of its gap on almost every day. A good range is calibrated (coverage close to the promise) and sharp (as narrow as possible).
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict and explain: a q90 forecast is 50. The real value is 40 on one day and 60 on another day. Which day has the larger pinball loss, and how many times larger?
Show the answer
The day of 60. Above the q90, each unit costs 0.9: 10 × 0.9 = 9. Below it, each unit costs 0.1: 10 × 0.1 = 1. The gaps are the same, 10 units, but the day above costs 9 times more, because a q90 should almost never be too low.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás says: 'Chronos-Bolt has the lowest pinball loss, so we use its 90% range for the safety stock.' What do you answer?
Show the answer
Its quantiles are the sharpest (pinball 0.099), but it predicts only the quantiles from 0.1 to 0.9. Its “90% range” is its 80% range, and it holds only 78.9% of the real values. A safety stock at its “q95” would run short about twice as often as planned. For a decision at the 90% or 95% level, use a method whose range keeps its promise at that level on the backtest. ETS does (90.5%).
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A good range is calibrated (its coverage is close to its promise) and sharp (as narrow as it can be). Coverage checks the first. It cannot catch a range that is too wide.
- The pinball loss scores each quantile: for a q90, a real value above costs 0.9 per unit and a value below costs 0.1. It rewards quantiles in the right place, and it makes a wide range pay for its width.
- On Larkfield, ETS and AutoARIMA keep both promises (83.5% and 90.5%; 83.0% and 90.2%) with a pinball loss of 0.110. Seasonal naive is too wide (0.133). Chronos-Bolt is the sharpest (0.099), but it holds only 78.9% and has no real 90% level.
- Conformal intervals build a range from past errors. With 4 windows, they made the ETS range worse: 68.9% and 75.6%. They need many windows, and each window is one more fit.
You can now tell a good range from a poor one. But Kenji still needs one number to order, and Grace one number of agents per hour. Next, you choose the quantile from the costs, and you compare the cost of each method’s decisions.