Measure the error that matters
- Lesson 10
- 40 min
- Module 3: Baselines and honest testing
Where we are
Our backtest gives us forecasts and real values at several cut-off dates. We still need one number that says which forecast is better. In this lesson, we compare MAE, RMSE, MAPE, WAPE, MASE and bias, see where each one breaks, and start the backtest table. We start with a baker who counts his mistakes.
Lesson 10 of 27 · Forecasting and Time Series: Predict What Comes NextMeasure the error that matters
1. Forecasts serve decisions
2. Read a time series
3. Baselines and honest testing
4. Statistical forecasters
5. Machine learning forecasters
6. Hard series and many series
7. Uncertainty and the decision
8. Run a forecast in production
What you will be able to do
Choose the error measure that matches the decision. Each measure answers a different question. Some of them break on Larkfield’s data: one gives an infinite score, and another can hide a forecast that is always too low.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It scores one week by hand with six measures, then MAPE and MAE on the 50 products, and prints the first rows of the backtest table. Its setup cells install the packages and download the Larkfield data, so it needs no local setup. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The rolling-origin backtest of the course (13 origins, 14 days), from Test like the future: backtesting.
- The four naive forecasts, from Naive forecasts.
- MAE and RMSE, from Regression metrics in the machine learning course. This lesson repeats them in short.
What you need on your computer
Nothing: the notebook runs in Colab or Kaggle.
Tested with Python 3.12, pandas 2.3.3 and statsforecast 2.1.1, in Jupyter on macOS.
A baker counts his mistakes
Section titled “A baker counts his mistakes”A baker bakes bread every morning from his forecast. In the evening, he counts the gap between the loaves he baked and the loaves he sold. Over a week, he wants one number that says how well he forecast. Which number?
- He can take the average gap in loaves: “I missed by 6 loaves a day.”
- He can say that big gaps hurt more. Ten loaves short on Saturday sends many customers away. One loaf short on ten days hardly matters.
- He can count the gap as a share of each day’s sales. But he sells 200 white loaves and 5 rye loaves a day. Five loaves is a small miss for white and a huge one for rye. And on a day when nobody bought rye, “the gap as a share of zero” makes no sense.
- He can add up all the gaps and divide by all the loaves sold that week.
- He can compare his gap with the gap of the simple rule “bake what you sold on the same day last week”.
- He can check if he is always too high or always too low. Always too high means stale bread every evening, even if the average gap looks small.
Each of these is a real error measure, with a name. Each one answers a different question. None of them is the “right” one for every case.
The problem: the same forecast, different verdicts
Section titled “The problem: the same forecast, different verdicts”At Larkfield, Tomás scores the seasonal naive forecast of the 50 products with MAPE, the “average percentage error”. Many reports use it. His notebook prints inf: infinite. Priya scores the same forecasts with WAPE and gets 51.3%. Omar, the product manager, asks: “Is the forecast infinitely bad, or does it miss half of the units?”
Both scores come from the same forecasts: one measure broke on Larkfield’s data. Before we fill the backtest table, we need to know what each measure does and where it fails.
Six measures on one week
Section titled “Six measures on one week”A garden lamp sold 4, 2, 0, 3, 5, 8, 6 in one week and 5, 3, 1, 3, 6, 10, 7 the next, Monday to Sunday. The seasonal naive forecast for the third week repeats the second week. The third week really sold 6, 2, 0, 4, 5, 12, 9. We made this small example up, so that you can check every number.
The error of a day is the real value minus the forecast. A positive error means the forecast was too low.
- Sold
- Forecast (seasonal naive)
Text description of the chart
Two lines with dots, for Monday to Sunday. Sold: 6, 2, 0, 4, 5, 12, 9. Forecast: 5, 3, 1, 3, 6, 10, 7. The lines stay within 1 lamp of each other from Monday to Friday. On Saturday and Sunday, the forecast is 2 lamps below the sales. On Wednesday, the sales are 0 and the forecast is 1.
Text description of the diagram
The diagram starts with a table of the lamp’s week. The real sales add up to 38 and the forecast to 35. The errors, real minus forecast, are +1, −1, −1, +1, −1, +2 and +2 (sum +3); their absolute values add up to 9. Below are four cards. MAE: 9 ÷ 7 = 1.29 lamps a day. RMSE: √(13 ÷ 7) = 1.36 lamps. WAPE: 9 ÷ 38 = 23.7%. MASE: 1.29 ÷ 1.00 = 1.29. A dashed orange box says that MAPE fails: Wednesday sold 0, and 1 ÷ 0 has no value.
The notebook computes all six measures with NumPy:
week1 = np.array([4, 2, 0, 3, 5, 8, 6])week2 = np.array([5, 3, 1, 3, 6, 10, 7])forecast = week2 # seasonal naive: the same weekday last weekreal = np.array([6, 2, 0, 4, 5, 12, 9])error = real - forecastprint("error:", error.tolist())with np.errstate(divide="ignore"): ape = np.abs(error) / real # a share of each day's real salesscale = np.abs(week2 - week1).mean() # the naive miss in the training weeksprint("MAE ", round(np.abs(error).mean(), 2))print("RMSE ", round(np.sqrt((error ** 2).mean()), 2))print("MAPE ", 100 * ape.mean(), "| without Wednesday:", round(100 * ape[real > 0].mean(), 1))print("WAPE ", round(100 * np.abs(error).sum() / real.sum(), 1))print("MASE ", round(np.abs(error).mean() / scale, 2), "| scale:", scale)print("bias ", round(100 * (forecast - real).sum() / real.sum(), 1))The output:
error: [1, -1, -1, 1, -1, 2, 2]MAE 1.29RMSE 1.36MAPE inf | without Wednesday: 25.1WAPE 23.7MASE 1.29 | scale: 1.0bias -7.9Now take the six measures one by one.
MAE and RMSE: misses in units
Section titled “MAE and RMSE: misses in units”MAE, the mean absolute error, is the mean size of a miss: 9 ÷ 7 = 1.29 lamps a day. It has the unit of the series, so it is easy to explain. RMSE, the root mean squared error, squares each error first. The two misses of 2 count as 4 each, so RMSE (1.36) is above MAE. Use RMSE when one large miss hurts more than several small ones. Regression metrics in the machine learning course explains both in detail.
Both have one weakness for Larkfield: their unit is the product’s unit, so you cannot compare them across products.
MAPE: breaks on a zero day
Section titled “MAPE: breaks on a zero day”MAPE, the mean absolute percentage error, divides each day’s miss by that day’s sales. On Monday, 1 ÷ 6 = 16.7%. On Wednesday, the lamp sold nothing, so the miss is 1 ÷ 0. That has no value, and NumPy gives inf, infinity. One zero day makes the whole MAPE infinite.
If you leave Wednesday out, MAPE says 25.1%. But now a day with 2 sales (Tuesday, 50% off) counts as much as a day with 12 (Saturday, 16.7% off). MAPE gives small days a large voice.
WAPE: misses as a share of sales
Section titled “WAPE: misses as a share of sales”WAPE, the weighted absolute percentage error, adds up all the misses first, then divides by all the sales: 9 ÷ 38 = 23.7%. A zero day adds its miss to the top and nothing to the bottom, so it causes no problem. Each day counts in proportion to its sales. That is the “weighted” in the name.
WAPE answers Kenji’s question: “Of all the units I needed, what share did the forecast miss?” It also works across products: add up the misses of all products, and divide by all their sales.
MASE: compared with a simple forecast
Section titled “MASE: compared with a simple forecast”MASE, the mean absolute scaled error, divides the forecast’s MAE by a scale. The scale is the MAE of the seasonal naive rule inside the training data. It says how far each day was from the same weekday one week before. For the lamp, week 2 minus week 1 is 1 lamp on every day, so the scale is 1.0. The forecast’s MAE is 1.29, so the MASE is 1.29 ÷ 1.0 = 1.29.
- Below 1: the forecast misses less than “the same weekday last week” usually missed in the past.
- Above 1: it misses more.
MASE has no unit, so you can average it across products. It also works with zero days, unless the training data is all zeros. The backtest helper computes the scale again at every origin, from the training days only.
Bias: too high or too low on average
Section titled “Bias: too high or too low on average”Bias adds up the forecasts minus the real values, and divides by the real values: (35 − 38) ÷ 38 = −7.9%. Negative means too low, positive means too high. A bias near zero does not mean a good forecast: large misses up and down can cancel. But a bias far from zero is a warning, even when the other measures look fine. A forecast that is always 8% too low runs out of stock week after week.
Larkfield: the measures on 50 products
Section titled “Larkfield: the measures on 50 products”MAPE breaks
Section titled “MAPE breaks”Priya backtests the seasonal naive forecast on the 50 products, as in the last lesson, and counts the zero days:
sales = load_sales(BACKTEST_PRODUCTS)fc = bt.backtest(sales, bt.seasonal_naive(7), h=14, origins=ORIGINS)err = fc["y"] - fc["yhat"]with np.errstate(divide="ignore", invalid="ignore"): ape = err.abs() / fc["y"]print("forecast days:", len(fc), "| with zero sales:", int((fc["y"] == 0).sum()))print("MAPE %:", 100 * ape.mean(), "| without the zero days:", round(100 * ape[np.isfinite(ape)].mean(), 1))print("WAPE %:", round(100 * err.abs().sum() / fc["y"].sum(), 1))forecast days: 9100 | with zero sales: 824MAPE %: inf | without the zero days: 77.1WAPE %: 51.3Of the 9,100 forecast days (50 products × 13 origins × 14 days), 824 sold nothing. So MAPE is infinite, as Tomás saw. Larkfield has many slow days. Even these 50 products, which sell at least 2 units a day on average, have a zero day about one time in eleven. WAPE has no problem with them: the forecast missed 51.3% of the units.
MAE does not compare across products
Section titled “MAE does not compare across products”Next, Priya computes the MAE of each product, and compares the slowest seller of the 50 with the best seller:
row = []for uid, g in fc.groupby("unique_id"): one = bt.score(g, sales, uid, season=7) row.append({"product": uid, "units a day": round(g["y"].mean(), 1), "MAE": round((g["y"] - g["yhat"]).abs().mean(), 2), "MASE": one["MASE"], "WAPE %": one["WAPE %"]})per = pd.DataFrame(row).sort_values("units a day")print(per.iloc[[0, -1]].to_string(index=False))product units a day MAE MASE WAPE % P029 2.1 1.87 1.090 87.7 P075 37.8 11.85 0.836 31.3Text description of the chart
Two columns. The slow seller P029 (2.1 units a day) has an MAE of 1.87. The best seller P075 (37.8 units a day) has an MAE of 11.85. Their MASE is 1.090 and 0.836, and their WAPE 87.7% and 31.3%.
The best seller has the larger MAE because it sells more, not because its forecast is worse. Its MASE and its WAPE are much better. If you averaged the MAE over 50 products, the best sellers would decide the average alone. So the backtest table uses MASE and WAPE, which have no unit.
The first rows of the backtest table
Section titled “The first rows of the backtest table”The course’s backtest table has one row per method and these columns:
| Column | What it says | Better |
|---|---|---|
MASE |
the mean of each product’s MAE ÷ its seasonal naive scale | lower; below 1 beats the naive scale |
WAPE % |
all misses ÷ all sales, every product and origin together | lower |
bias % |
(all forecasts − all sales) ÷ all sales | closer to 0 |
pinball |
a score for forecast ranges, from Module 7 | lower |
fit s |
the seconds the method needed, all origins together | lower |
cost $ |
the cost of Kenji’s 14-day order, per order | lower |
The cost column needs the course’s cost assumptions. For each product and origin, the order is the forecast of the 14 days added up. A unit short loses its margin: 35% of the price. A unit left over is sold off at the end of the season, with a loss of 25% of the price. These are assumptions, not Larkfield’s accounts. Module 7 explains them and turns them into a better order.
You ran these four rows with bt.score() and bt.show() in Naive forecasts. Section 4 of the notebook runs the same cell again, and it keeps each method’s forecasts in forecasts. bt.show() prints the table with the best MASE first:
method horizon MASE WAPE % bias % pinball fit s cost $moving average (28 days) 14 0.875 43.5 -0.1 0.03 19444 seasonal naive 14 1.034 51.3 -0.5 0.04 16739 naive 14 1.065 53.2 7.4 0.02 28261 drift 14 1.075 53.8 7.8 0.03 28570The fit times change a little from run to run. Every later module adds its rows to this table.
Text description of the chart
A column chart with four columns. WAPE: moving average of 28 days 43.5%, seasonal naive 51.3%, naive 53.2%, drift 53.8%.
Text description of the chart
A column chart with four columns. Cost per order: moving average of 28 days 19,444 dollars, seasonal naive 16,739, naive 28,261, drift 28,570. The seasonal naive forecast has the lowest cost although its WAPE is higher than the moving average’s.
Read the table one column at a time:
- MASE: only the moving average is below 1. The other three miss more than the seasonal naive scale. Even the seasonal naive forecast itself is above 1 (1.034). Its scale compares each day with the day exactly one week before. Its forecast for the second week repeats a day two weeks old, which is harder.
- WAPE: the same order as MASE here. That is not always so: MASE averages products equally, and WAPE weighs them by their sales.
- bias: naive and drift are too high by 7% to 8%. They repeat the last day of the history, a Sunday, which sells more than an average day.
- cost: the seasonal naive forecast is cheapest, although it is second by WAPE. Its daily misses are larger, but they cancel more over the 14 days of an order.
The lowest error is not always the lowest cost. Kenji pays for the order, not for the daily misses. Module 7 makes this the main question of the course.
Which measure, when
Section titled “Which measure, when”| Your question | Use | Do not use |
|---|---|---|
| How many units do we miss, in total, across products? | WAPE | MAPE (breaks on zero days, favours small days) |
| Does this method beat a simple forecast, on average over products? | MASE | MAE (its unit differs per product) |
| How big is a typical miss for one product, in its unit? | MAE | WAPE alone (a share hides the size) |
| Do large misses hurt much more than small ones? | RMSE | MAE |
| Is the forecast too high or too low? | bias, always next to WAPE or MASE | bias alone (misses up and down cancel) |
| What does the forecast cost the business? | the cost of the decision | any error measure alone |
Every measure costs almost nothing to compute. The real cost is choosing the wrong one. A team that reports MAPE drops the zero days without knowing it. A team that reports MAE across products only watches its best sellers. Agree on the measures before you compare methods, write them next to every number, and keep the same ones for the whole project.
Common mistake
Dropping the zero days to make MAPE work
Symptom: Tomás sees inf and filters out the days with zero sales. Now MAPE says 77.1% for the seasonal naive forecast. He compares methods by this number.
Cause: the filter removed 824 of the 9,100 forecast days. A forecast of 3 on a day that sold 0 is a real miss, and Kenji pays for the left-over stock. MAPE also gives a day with 1 sale the same weight as a day with 100. So a method that is good on small days and poor on big days looks better than it is. And methods that forecast low get a better MAPE, because a miss on a small real value is a large percentage.
Fix: use WAPE for “how much did we miss” and MASE for “do we beat the naive forecast”. Both keep the zero days. If a report must have a percentage, give WAPE.
Consequence in AI work: MAPE is common in dashboards and in model reports. On data with zero days, it either breaks or quietly leaves out the days that are hardest to forecast. Slow sellers, new products and help-desk hours at night all have zero days.
Guided practice: six measures in the notebook
Section titled “Guided practice: six measures in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell. The data cell ends with the
Ready:line. - In section 1, predict which measure fails on Wednesday. Then run the cell. You see MAPE
infand WAPE23.7. - In section 2, predict how many forecast days have zero sales. Then run the cell. You see
824. - In section 3, run the cell. You see P029 with MAE
1.87and P075 with MAE11.85. - In section 4, predict which baseline has the lowest cost. Then run the cell. You see four rows of the backtest table.
- In section 5, compute the bias of the naive forecast in the Your turn cell.
- Run the check cell. It prints
Check passed.
Your turn: the bias of the naive forecast
Section titled “Your turn: the bias of the naive forecast”forecasts["naive"] holds every forecast of the naive method, with the real value y and the forecast yhat. Before you compute anything, predict: is the naive forecast too high or too low on average? Then compute its bias % yourself: the forecasts minus the real values, added up, divided by the real values added up, times 100. Round it to 1 decimal and put it in bias_naive.
Show the answer
f = forecasts["naive"]bias_naive = round(float(100 * (f["yhat"] - f["y"]).sum() / f["y"].sum()), 1)print(bias_naive)7.4The naive forecast is 7.4% too high: over 13 orders, it would order 7.4% more units than Larkfield sold. It repeats the last day of the history, a Sunday, for 14 days. The value equals the bias % column of the table, because bt.score() uses the same formula.
Knowledge check
Section titled “Knowledge check”Check your understanding
The lamp's forecast week has the errors +1, −1, −1, +1, −1, +2, +2 and the real sales 6, 2, 0, 4, 5, 12, 9. Compute WAPE and bias, and say why MAPE cannot be computed.
Show the answer
- WAPE: the absolute errors add up to 9 and the sales to 38, so WAPE is 9 ÷ 38 = 23.7%.
- Bias: the forecasts add up to 35, so the bias is (35 − 38) ÷ 38 = −7.9%: the forecast is too low.
- MAPE divides each day’s miss by that day’s sales. Wednesday sold 0, so its share is 1 ÷ 0, which has no value. One zero day makes MAPE infinite.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás wants to rank the 50 products by forecast quality with MAE. What goes wrong, and what do you use instead?
Show the answer
MAE has the unit of each product. A best seller has a larger MAE only because it sells more. P075 sells 37.8 units a day and has an MAE of 11.85. P029 sells 2.1 units a day and has an MAE of 1.87. Yet the forecast of P075 is much better. Use MASE, which divides each product’s MAE by its own naive scale (P075 0.836, P029 1.090). Or use WAPE, which divides by its own sales (31.3% against 87.7%).
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The seasonal naive forecast has a bias of −0.5%. Omar says: 'Almost zero bias, so the forecast is almost perfect.' Is he right?
Show the answer
No. Bias adds up the misses with their signs, so misses up and misses down cancel. The seasonal naive forecast misses 51.3% of the units (WAPE), and its MASE is 1.034. It is too high on some days and too low on others by about the same amount. A bias near zero only says that the forecast is not too high or too low on average. Always read bias next to WAPE or MASE.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- MAE and RMSE measure misses in the product’s unit; RMSE counts large misses more. They do not compare across products.
- MAPE divides by each day’s sales. On Larkfield, 824 of 9,100 forecast days sold nothing, so MAPE is infinite. Dropping those days hides real misses.
- WAPE adds up all misses and divides by all sales: 51.3% for the seasonal naive forecast. It works with zero days and across products.
- MASE compares a forecast with the seasonal naive scale; below 1 is better. Bias says too high or too low, and must be read next to WAPE or MASE.
- The backtest table starts with four baselines. The 28-day moving average has the best MASE (0.875) and WAPE (43.5%). The seasonal naive forecast has the lowest cost (16,739 dollars).
You can now score any forecast with the measures that match the decision, and you have the baselines to beat. But the baselines only repeat or average the past, and the best one still misses 43.5% of the units. Next, you build a forecast that weighs recent days more and learns the trend and the season.