Regression metrics

What you will be able to do

Measure how far a predicted number is from the real one, and choose the summary that matches what a wrong forecast costs. Two forecasts with the same average miss can be very different for the people who plan with them.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson, with the instructions. Its setup cells install pandas and scikit-learn and download the practice data, so it works without the local project. Read the explanations here, and run the code there.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• What regression is: a model that predicts a number, from Supervised learning.
• The daily ticket counts in daily_tickets.csv, from Choose the learning task.
• Why we judge a forecast on data that it did not learn from, from Train, validation and test sets: Three sets, three jobs.
• The mean, from Describe a dataset: The mean and the median, and a squared error, from Loss and gradients, both in Essential Mathematics and Statistics for AI.

What you need on your computer (only for the local route; the notebook needs none of it)

• The project folder ~/projects/ticket-model with its .venv (pandas 3.0.6, scikit-learn 1.9.1, matplotlib 3.11.2) and the data in data/. You made them in Define a baseline: Set up your project. If a data file is missing or changed, run python get_data.py --reset. The last line must be Ready: train 16939 rows (1983 escalated), valid 1179 (183), test 1002 (157).
• To run the notebook locally, open it in VS Code with the Jupyter extension and the .venv kernel, as in Define a baseline: Open a notebook in VS Code.

Tested on macOS with Python 3.14.6, pandas 3.0.6, scikit-learn 1.9.1 and matplotlib 3.11.2. The Windows commands are not tested. The notebook was tested in Jupyter.

Two kinds of answer, two kinds of metric

A metric is a number that says how good the predictions are. You compute it by comparing each prediction with the real answer, the label. Labels are what make this possible in supervised learning: for past cases, we know what really happened.

Which metric makes sense depends on the kind of answer. A regression model gives a number, such as 30 tickets tomorrow. When 40 tickets arrive, you can subtract: the forecast missed by 10 tickets. A classification model gives a class, such as “escalated” or “not escalated”. You cannot subtract “not escalated” from “escalated”, so the next lesson counts outcomes instead. This lesson is about numbers.

What the model gives decides how you measure it. This lesson covers the top row; the next two lessons cover the other two.

Text description of the diagram

The diagram starts with a question: what does the model give? Three paths follow. A number, for example real 40 and forecast 30, leads to a subtraction: error = real − forecast = 10 tickets. That path ends at the regression metrics: mean error, MAE, RMSE, MAPE and R². A class, escalated or not, leads to counting four outcomes (TP, FP, FN, TN), and then to the classification metrics: accuracy, precision, recall and F1. A score ranks the tickets. It leads to choosing a cut-off, or to summarising every cut-off, and then to the trade-off metrics: precision@k, recall@k, the PR curve, average precision and ROC AUC. A dotted arrow shows that each cut-off of a score gives a class, so the counts apply again.

A forecast that misses

A delivery app says your food will arrive in 30 minutes, and it arrives after 40. The app missed by 10 minutes. Yesterday it said 30 and the food came in 28: a miss of 2 minutes the other way. A weather forecast says 20 degrees, and the day reaches 22. In each case, you compare a predicted number with the real number, and the difference has a size and a unit.

That difference is the error of one prediction:

error = real value − predicted value

The sign tells you the direction. A positive error means the forecast was too low: more tickets came than predicted. A negative error means the forecast was too high. The unit is the unit of the target: minutes, degrees, or here, tickets per day.

At Larkfield, Omar, the product manager, plans the agents’ shifts the evening before. He needs to know how many tickets will arrive tomorrow. One forecast and one day give one error. But Omar plans every day, so he needs one number that describes the errors of many days. That number is a regression metric. There are several, because errors can be summarised in several ways, and each way answers a different question.

Five days, two forecasts

Here are five days of real ticket counts, from Thursday to Monday. Two forecasts were made the evening before each day. Forecast A says 30 on a weekday and 15 at the weekend. Forecast B says 26 every day.

Day	Real	Forecast A	Error A	Forecast B	Error B
Thu	31	30	+1	26	+5
Fri	30	30	0	26	+4
Sat	14	15	−1	26	−12
Sun	13	15	−2	26	−13
Mon	40	30	+10	26	+14

Look at the errors before you read on. Forecast A is close on four days and far off on Monday. Forecast B is far off on every day. Now see how each metric describes this.

Why not just average the errors?

The first idea is to take the mean of the errors. For A: (1 + 0 − 1 − 2 + 10) ÷ 5 = +1.6 tickets. For B: (5 + 4 − 12 − 13 + 14) ÷ 5 = −0.4 tickets.

By this number, B looks better than A. It is not. B misses by 12 or more on three days, but its positive and negative errors cancel. A forecast that is 20 too high one day and 20 too low the next has a mean error of 0.

So the mean of the signed errors cannot tell you how large the misses are. It answers a different question, and that question is useful: is the forecast too low or too high on average? This number is the mean error, also called the bias. A mean error near 0 means the misses go both ways. A clearly positive mean error means the forecast is too low most of the time: Omar would plan too few agents again and again. Always check the mean error, but never use it alone.

MAE: the average size of a miss

To measure the size of the misses, drop the signs first. The size of an error without its sign is its absolute error: −12 becomes 12. Then take the mean.

• A: (1 + 0 + 1 + 2 + 10) ÷ 5 = 2.8 tickets
• B: (5 + 4 + 12 + 13 + 14) ÷ 5 = 9.6 tickets

This is the mean absolute error (MAE). It answers: “on a typical day, how many tickets is the forecast off?” It is in the target’s unit, so anyone can read it: “forecast A is off by about 3 tickets a day”. MAE counts every ticket of error the same: a miss of 10 counts exactly ten times as much as a miss of 1.

RMSE: big misses count more

Now look at A again. Its MAE, 2.8, is an average of four small misses and one large one. But for Omar, one large miss is much worse than a few small ones. A day with 10 more tickets than planned leaves the agents with a queue that lasts into the next day. Two days with 1 extra ticket each change nothing.

To give large misses more weight, square each error before you take the mean. A square grows fast: 1 stays 1, 2 becomes 4, but 10 becomes 100. Then take the square root of the mean, to get back to tickets.

• A: the squares are 1, 0, 1, 4 and 100. Their mean is 106 ÷ 5 = 21.2, and √21.2 = 4.60 tickets.
• B: the squares are 25, 16, 144, 169 and 196. Their mean is 110, and √110 = 10.49 tickets.

This is the root mean squared error (RMSE). Its name gives the steps in reverse order: the root of the mean of the squared errors.

Forecast A on five days. Top: each error as a bar, up when the forecast was too low (blue), down when it was too high (orange). Bottom: each error squared, drawn as a square whose side is the error. Squaring removes the sign and makes Monday's miss dominate.

Text description of the chart

Five columns, one for each day from Thursday to Monday. In the top row, each error is a bar from a zero line. Thursday: a short bar up, +1. Friday: no bar, 0. Saturday: a short bar down, −1. Sunday: a bar down, −2. Monday: a tall bar up, +10, ten times Thursday’s bar. In the bottom row, each error is squared and drawn as a square with that side. Thursday and Saturday: tiny squares with area 1. Friday: nothing. Sunday: a small square, 4. Monday: a large square, 100. Monday’s square is 25 times Sunday’s, while its bar is only 5 times as tall. The squares of the other four days together are 6, so Monday gives 100 of the 106.

The picture shows why RMSE is larger than MAE for forecast A: Monday gives 100 of the 106 in the sum of squares. Without Monday, the errors of A are 1, 0, 1 and 2: MAE is 1.0 and RMSE is 1.22, almost the same. RMSE is much larger than MAE when a few misses are large. So the gap between the two numbers is itself a signal: a large gap means some days are very wrong.

MAPE: the miss as a share of the real value

A miss of 5 tickets is not always the same. On a quiet Sunday with 12 tickets, a forecast of 17 is off by 5, which is 42% of the real number. On a busy Monday with 40 tickets, a forecast of 35 is also off by 5, but that is only 12.5%. When the real values have very different sizes, a percentage can be fairer than a count.

The mean absolute percentage error (MAPE) divides each absolute error by the real value, and takes the mean of these percentages. For forecast A:

Day	Real	Absolute error	Percentage
Thu	31	1	3.2%
Fri	30	0	0%
Sat	14	1	7.1%
Sun	13	2	15.4%
Mon	40	10	25.0%

MAPE of A = (3.2 + 0 + 7.1 + 15.4 + 25.0) ÷ 5 = 10.2%. MAPE of B is 50.0%: it is off by 85.7% on Saturday and 100% on Sunday.

MAPE has one serious problem: it breaks near zero. If a day has 2 real tickets and the forecast says 4, the miss is only 2 tickets, but it is 100%. If a day has 0 tickets, MAPE must divide by 0, and it has no value. So use MAPE only when the real values are never close to zero. In May and June, Larkfield’s quietest day has 11 tickets, so MAPE works here. For a product that sells 0, 1 or 2 units a day, it does not.

R²: better than the mean?

The last metric compares the forecast with a very simple one: the mean of the real values. For the five days, the real mean is 25.6 tickets. A forecast that says 25.6 every day is a reference. The R² (say “R squared”) asks how much better a forecast is than that reference.

To compute it, compare two sums of squares. The forecast’s squared errors add up to 106 for A. The reference’s squared errors, the real values minus 25.6, add up to 549.2. Then R² = 1 − 106 ÷ 549.2 = 0.807. People say that A “explains about 81% of the variation” from day to day.

• R² = 1 means a perfect forecast: no error at all.
• R² = 0 means no better than the mean. Forecast B says 26 every day, almost the mean, and its R² is −0.001, about 0.
• R² below 0 is possible. It means the forecast is worse than the mean of the real values. You see this in the worked example.

R² is common in reports and has no unit, which makes it easy to compare between datasets. But it hides what Omar needs. An R² of 0.8 can come from small misses on days that vary a lot. It can also come from large misses on days that vary even more. It does not say how many tickets the forecast is off. Report it next to MAE or RMSE, not alone.

The formulas

You already know every step from the numbers above. Here they are in symbols, with y as the real value, ŷ (“y hat”) as the forecast, and n days:

$$\text{mean error} = \frac{1}{n}\sum (y - \hat{y}) \qquad \text{MAE} = \frac{1}{n}\sum |y - \hat{y}| \qquad \text{RMSE} = \sqrt{\frac{1}{n}\sum (y - \hat{y})^2}$$

$$\text{MAPE} = \frac{100\%}{n}\sum \left|\frac{y - \hat{y}}{y}\right| \qquad R^2 = 1 - \frac{\sum (y - \hat{y})^2}{\sum (y - \bar{y})^2}$$

Here ȳ (“y bar”) is the mean of the real values. scikit-learn computes all of them: mean_absolute_error, mean_squared_error (take its square root for RMSE), mean_absolute_percentage_error (it returns a share, so multiply by 100) and r2_score. The mean error is one line of pandas: (real - forecast).mean().

Choose the metric

Every metric above is correct. They differ in what they treat as bad. So the choice does not come from the data; it comes from what a wrong forecast costs the people who use it. Ask two questions. First: is a miss of 10 just ten times as bad as a miss of 1, or much worse? Second: do the real values have very different sizes?

If every unit of error costs about the same, use MAE. A delivery app that is late by 4 minutes annoys a customer about twice as much as one that is late by 2. If a large miss is much worse than several small ones, use RMSE. A power company that under-predicts the demand by a large amount risks a blackout, and a few small misses cost almost nothing. If you compare forecasts across very different sizes, such as small and large shops, use MAPE. Do not use it when a real value can be close to zero. And always check the mean error. A forecast that is too low almost every day causes a problem that MAE and RMSE do not show directly.

Situation	What a wrong forecast costs	Main metric	Check next to it
A delivery app’s arrival time	Each minute of delay annoys about the same	MAE	Mean error: is the app always too optimistic?
Electricity demand for tomorrow	A large under-forecast can cause a blackout	RMSE	The largest miss
Sales of 500 shops, small and large	The same miss matters more in a small shop	MAPE	MAE in units, for the largest shops
Stock to order for a warehouse	Ordering too little every week empties the shelves	Mean error	MAE
Larkfield’s tickets per day, for shifts	A very busy day that was not planned overloads the agents	RMSE	Mean error and MAE
A report to compare models on one dataset	Needs one number without a unit	R²	MAE or RMSE in the target’s unit

For Larkfield, Priya, the data scientist, chooses RMSE as the main number, because large misses hurt the agents most. She reports MAE next to it, so that Omar can read “off by about N tickets a day”. She also reports the mean error, so that he knows if the forecast is too low.

Worked example: Priya’s forecasts for May and June

No model exists yet in this course. That is fine: a forecast does not need machine learning. Priya makes three forecasts that anyone could make with a spreadsheet. Each one learns only from the days before 1 May 2026, the training days. She evaluates them on the 61 days of May and June 2026, which no forecast has seen.

1. Train mean: the mean of all training days, every day.
2. Weekday or weekend mean: one mean for weekdays, one for Saturdays and Sundays.
3. Same weekday and month: the mean of the training days with the same weekday in the same month. For a Monday in May, that is the mean of the Mondays of May 2025.

Each code block is a cell of the lesson notebook, where the setup cells have already loaded pandas as pd and numpy as np. First, the split:

daily = pd.read_csv("data/daily_tickets.csv", parse_dates=["date"])
train_days = daily[daily["date"] < "2026-05-01"]
eval_days = daily[daily["date"] >= "2026-05-01"].copy()
print(len(train_days), "training days,", len(eval_days), "days to evaluate")
print("mean per day: training", round(train_days["tickets"].mean(), 2), "| May-June", round(eval_days["tickets"].mean(), 2))

The output:

Output

669 training days, 61 days to evaluate
mean per day: training 25.32 | May-June 37.59

May and June are much busier than the training days. They are Larkfield’s garden season, and the shop also has more customers each month. Keep that in mind. Next, the three forecasts:

eval_days["train mean"] = train_days["tickets"].mean()
weekend_means = train_days.groupby("is_weekend")["tickets"].mean()
eval_days["weekday or weekend mean"] = eval_days["is_weekend"].map(weekend_means)
same_means = train_days.groupby(["month", "weekday"])["tickets"].mean()
eval_days["same weekday and month"] = [same_means[(month, weekday)]
                                       for month, weekday in zip(eval_days["month"], eval_days["weekday"])]
print(weekend_means.round(2).to_dict())

Output

{0: 29.99, 1: 13.54}

A weekday gets 29.99 and a weekend day 13.54. Then a function that computes the five metrics for one forecast, and a table with one row per forecast:

from sklearn.metrics import mean_absolute_error, mean_absolute_percentage_error, mean_squared_error, r2_score

def regression_report(real, forecast):
    """Five numbers that describe the errors of a forecast."""
    errors = real - forecast
    return {"mean error": round(float(errors.mean()), 2),
            "MAE": round(mean_absolute_error(real, forecast), 2),
            "RMSE": round(mean_squared_error(real, forecast) ** 0.5, 2),
            "MAPE %": round(100 * mean_absolute_percentage_error(real, forecast), 1),
            "R2": round(r2_score(real, forecast), 3)}

forecasts = ["train mean", "weekday or weekend mean", "same weekday and month"]
print(pd.DataFrame({name: regression_report(eval_days["tickets"], eval_days[name]) for name in forecasts}).T)

Output

                         mean error    MAE   RMSE  MAPE %     R2
train mean                    12.27  14.95  17.27    38.3 -1.018
weekday or weekend mean       12.45  12.54  14.08    32.2 -0.340
same weekday and month         5.00   6.13   7.83    16.4  0.586

Read the table one column at a time:

• Mean error. All three forecasts are too low on average. The first two miss by about 12 tickets a day in the same direction. They learned from a quieter time and cannot know that May and June are busier.
• MAE. “Same weekday and month” is off by about 6 tickets on a typical day, less than half of the others. It knows that May is a busy month, because May 2025 was busy too.
• RMSE. Each RMSE is a little larger than its MAE. The best forecast has RMSE 7.83: some days miss by much more than 6 tickets. Its largest miss is 24.2 tickets, and 11 of the 61 days miss by more than 10.
• MAPE. The best forecast is off by 16.4% of the real count on a typical day.
• R². The first two forecasts have an R² below zero. They are worse than the mean of May and June, 37.59, which they could not know. The best forecast has R² 0.586.

The chart shows the real counts and the three forecasts, day by day.

• Real tickets
• Train mean (dashed)
• Weekday or weekend mean
• Same weekday and month

Real tickets per day in May and June 2026, and three forecasts made from the days before May. The two simple forecasts are below the real counts on almost every day.

Text description of the chart

A line chart of 61 days, from 1 May to 30 June 2026, with tickets per day from 0 to 60 on the vertical axis. The real counts go up and down every week: mostly 36 to 52 on weekdays and 15 to 26 at weekends, with a highest day of 60 and a lowest of 11. The train mean is a flat dashed line at 25.3, below the real counts on most days. The weekday or weekend forecast jumps between 30.0 on weekdays and 13.5 at weekends. It has the right weekly shape, but it is below the real counts on 60 of the 61 days. The same-weekday-and-month forecast is between 17.5 and 42.25. It follows the weekly shape and the level of May and June more closely. But it is still below the real counts on 50 of the 61 days.

Priya writes a short note for Omar: “The best simple forecast is off by about 6 tickets a day. On an average day, it is about 5 tickets too low, because the shop keeps growing. Plan with it, but add a few agents.” The numbers tell her what to fix next: the bias. Module 4 builds a regression model, and you meet these numbers again there.

Guided practice: the metrics by hand and in code

Use the notebook, sections 1 to 3. You compute the five metrics for the five days, first by hand and then with scikit-learn.

1. Open the notebook from the box at the top of this page, in Colab, Kaggle or VS Code.
2. Run the setup cells. You should see pandas 3.0.6, scikit-learn 1.9.1 and the Ready: line.
3. In section 1, compute the mean error, MAE and RMSE of forecast A on paper first. Then run the cell. You should see mean error 1.6, MAE 2.8 and RMSE 4.6.
4. In section 2, write the MAPE of forecast A with numpy. Put it in mape_a, as a percentage rounded to 1 decimal.
5. Run the check cell. It prints Check passed: mape_a is 10.2. when your number is right.
6. In section 3, run regression_report() on forecast B. You should see a mean error of -0.4, MAE 9.6 and R² -0.001.
7. Write one sentence for Omar about forecast B. Use the mean error and the MAE.

Show the answer

real = np.array([31, 30, 14, 13, 40])
forecast_a = np.array([30, 30, 15, 15, 30])
mape_a = round(float(100 * np.mean(np.abs(real - forecast_a) / real)), 1)
print(mape_a)

Output

10.2

For Omar: “Forecast B is right on average: its mean error is −0.4. But it is off by almost 10 tickets on a typical day. It is too high at weekends and too low on weekdays, and the two cancel in the mean error.” The mean error alone would make B look good. MAE shows the size of the misses.

Your turn: the last four weeks

Change one thing: the training days that a forecast uses. The three forecasts above are too low, because the shop grows. A forecast that looks only at recent days may follow the growth. The new forecast is the mean of the same weekday over the last 4 weeks before May (3 to 30 April 2026). For a Monday in May, it is the mean of the four Mondays of April.

Before you run section 4 of the notebook, predict. Will the mean error be closer to 0 than the 5.00 of “same weekday and month”? Will MAE be lower? Then compute the column same weekday, last 4 weeks and its regression_report().

Show the answer

last_4_weeks = train_days[train_days["date"] >= "2026-04-03"]
weekday_means = last_4_weeks.groupby("weekday")["tickets"].mean()
eval_days["same weekday, last 4 weeks"] = eval_days["weekday"].map(weekday_means)
print(regression_report(eval_days["tickets"], eval_days["same weekday, last 4 weeks"]))

Output

{'mean error': 1.65, 'MAE': 5.08, 'RMSE': 6.32, 'MAPE %': 15.1, 'R2': 0.73}

The mean error falls from 5.00 to 1.65: April was already busy, so the bias is much smaller. MAE falls from 6.13 to 5.08 and RMSE from 7.83 to 6.32. Recent data helps when the level changes over time. There is a cost. A forecast from only 4 weeks moves with every unusual week, such as a holiday or an outage, because it has few days to average.

Common mistake

A small MAE that hides a bias

Symptom: Tomás, the AI engineer, compares the forecasts by MAE only. He reports: “Same weekday and month is off by 6.13 tickets a day. That is good enough.” Omar plans the May shifts with it. After two weeks, Grace, the support lead, says that the queue is long almost every evening.

Cause: MAE measures the size of the misses, not their direction. This forecast is too low on 50 of the 61 days, by 5.00 tickets on average. A forecast that is sometimes too low and sometimes too high is a small problem for staffing. A forecast that is too low almost every day leaves the team short every day. The mean error shows it at once; MAE does not.

errors = eval_days["tickets"] - eval_days["same weekday and month"]
print("mean error:", round(errors.mean(), 2), "| days too low:", (errors > 0).sum(), "of", len(errors))

Output

mean error: 5.0 | days too low: 50 of 61

Fix: report the mean error next to MAE or RMSE every time, and count the days that are too low and too high. If the bias is large, find its cause before you trust the forecast. Here the cause is growth, and a forecast from recent weeks reduces the bias to 1.65.

Consequence in AI work: a model that is biased in one direction does quiet harm. A demand forecast that is always low empties shelves; a risk model that is always low under-prices insurance. The usual summary metric can look good while every error points the same way.

See it end to endForecast bike-share demandCase study

The bike-share case study chooses MAE as its main metric and reports RMSE for the peaks, for a manager who plans bikes per hour.

See it end to endEstimate house pricesCase study

The house-price case study judges a valuation model by its percentage error.

Knowledge check

Check your understanding

Two forecasts are tested on the same four days. Forecast A misses by 3, 3, 3 and 3 tickets. Forecast B misses by 0, 0, 0 and 12 tickets. Compute MAE and RMSE for both. A very busy day that was not planned is the worst case for Larkfield. Which forecast do you recommend?

Show the answer

Both have MAE = 12 ÷ 4 = 3 tickets. Forecast A has RMSE = √((9 + 9 + 9 + 9) ÷ 4) = 3. Forecast B has RMSE = √(144 ÷ 4) = 6. MAE cannot tell them apart; RMSE can. Recommend A: forecast B has one day that is 12 tickets off, and that day overloads the agents. When large misses are the worst case, choose with RMSE, and show MAE next to it.

Check your understanding

A shop forecasts daily sales for 200 products. Some products sell 0 or 1 unit on most days. A colleague wants to compare forecasts with MAPE. What goes wrong, and what do you suggest?

Show the answer

MAPE divides each miss by the real value. On a day with 1 real unit, a forecast of 2 is a 100% error. On a day with 0 units, MAPE divides by zero and has no value (or becomes very large). So MAPE is unstable for products with sales close to zero. Use MAE in units instead, or compute MAPE only for products whose sales are never close to zero.

Check your understanding

Predict: a forecast has a mean error of +8 tickets and an MAE of 8 tickets. What can you say about its errors, without looking at them?

Show the answer

Every error is positive or zero: the forecast is too low on every day, or exact. The mean error can equal MAE only when no error is negative, because a negative error lowers the mean error but raises MAE. So the forecast is biased: it always predicts too few tickets, by 8 on average. Adding about 8 to it would remove most of the error, if the bias stays the same.

Recap

• A regression model predicts a number, so each prediction has an error: real − predicted, with a sign and a unit.
• The mean error (bias) shows the direction: positive means the forecast is too low on average. Errors with different signs cancel in it, so it never measures the size of the misses.
• MAE is the typical size of a miss. RMSE squares first, so a few large misses raise it much more. MAPE is the miss as a percentage, useful across sizes and useless near zero. R² compares with the mean of the real values, and can be below zero.
• Choose the metric from what a wrong forecast costs. For Larkfield’s shifts: RMSE, with MAE and the mean error next to it.
• On May and June, the best simple forecast is the same weekday and month. It has MAE 6.13, RMSE 7.83 and a mean error of +5.00: it is too low on 50 of 61 days.

Next, the escalation question. Its answer is a class, not a number, so you count outcomes instead of subtracting.

Next: Classification metrics