Watch the forecast
- Lesson 26
- 40 min
- Module 8: Run a forecast in production
Where we are
Our job writes a forecast every week and keeps it. Nobody looks at it yet after the week ends. In this lesson, we compare last week's forecast with what happened, add a bias alarm, and decide when to retrain and when to change the method. Think first of a kitchen scale that is always a little wrong.
Lesson 26 of 27 · Forecasting and Time Series: Predict What Comes NextWatch the forecast
1. Forecasts serve decisions
2. Read a time series
3. Baselines and honest testing
4. Statistical forecasters
5. Machine learning forecasters
6. Hard series and many series
7. Uncertainty and the decision
8. Run a forecast in production
What you will be able to do
Find out, every week, whether last week’s forecast was right, and notice when it is wrong in the same way again and again. Then decide what to change: the data, the fit, or the method.
Before you startWhat to know and to have ready
What you need to know
- Your job folder
forecast-jobwith this course’s storeruns/, from The forecasting job. - Bias: the forecast total minus the real total, as a share of the real total. You met it in Measure the error that matters.
- The yearly season, from Trend, season and noise and Several seasons and long series.
What you need on your computer
The same as in the last lesson: the job’s virtual environment, active, in forecast-job. The replays in this lesson take about 45 seconds each.
A kitchen scale that is always wrong
Section titled “A kitchen scale that is always wrong”You weigh flour for a cake. The recipe says 500 grams. One day the cake comes out a little dry. Was the scale wrong? Maybe. Maybe you spilled some flour, or the oven was hot. One bad cake tells you little.
Now the scale shows 450 grams every time you put a 500-gram packet on it. Every day, it is low, and always by about the same amount. That is no longer bad luck. The scale has a bias: an error that keeps the same direction. You do not fix it by weighing more carefully. You fix the scale, or you buy another one.
A forecast can be like that scale. A single bad week is normal. A forecast that is too low week after week in the same category needs a person to look at it.
The problem: nobody checks last week’s plan
Section titled “The problem: nobody checks last week’s plan”The job writes a plan every Monday and keeps it. But the plan is only a guess about the next 14 days. A week later, the sales of the first 7 days are known. Unless someone compares the two, a wrong forecast stays wrong.
Look at garden furniture in the spring of 2025. Each point is the job’s forecast for the first 7 days of one week’s plan, and what was sold in those 7 days:
- Units sold in the 7 days
- The job’s forecast for those 7 days
Text description of the chart
The chart has 11 weeks, from the plan of 3 March to the plan of 12 May 2025. In the first two weeks, the forecast and the sales are close: 87 against 88, then 96 against 111. From the week of 17 March, the sales rise fast: 203, 392, 608 units. The forecast follows late: 112, 162, 358. In April, the two lines come close twice (a forecast of 520 against 452 sold on 7 April, and 600 against 614 on 21 April). In May, the sales jump again, to 1,600 and 2,559 units, and the forecast stays about 40% lower: 920 and 1,532.
In the week of 24 March, the job forecast 162 units and 392 were sold. An order made from that forecast is too small. The week after, the forecast was too low again. Each Monday, the job refit its model on the newest sales, but it still trailed the season.
The idea: check every plan, and alarm on bias that repeats
Section titled “The idea: check every plan, and alarm on bias that repeats”The check is simple. For each category, take the plan of last week and add up its forecast for the 7 days. Add up the units sold on the same 7 days. Then:
bias % = 100 × (forecast − sales) ÷ sales
A negative bias means the forecast was too low: a risk of empty shelves. A positive bias means it was too high: left-over stock.
Here is the worked example for garden furniture, from the job’s file runs/2025-03-24/check.csv:
| Week of the plan | Forecast (7 days) | Sold | Forecast − sold | Bias |
|---|---|---|---|---|
| 17 March | 111.9 | 203 | −91.1 | −91.1 ÷ 203 = −44.9% |
| 24 March | 162.2 | 392 | −229.8 | −229.8 ÷ 392 = −58.6% |
Two weeks in a row, the forecast was much too low. That is the kitchen scale that always shows less.
A bias alarm turns this into a rule that the job checks every week. Our job’s rule has three parts:
Text description of the diagram
On the left, two boxes: last week’s plan, its forecast for the first 7 days, and what was sold in those 7 days. Both feed a box: bias per category, (forecast minus sales) divided by sales, for example −45% too low or +81% too high. Then three questions in a row. First: 20% or more off? Second: off the same way the week before? Third: 200 units or more sold in both weeks? If all three answers are yes, the result is BIAS ALARM: look for the cause before the next order. If any answer is no, the result is no alarm.
Each part has a reason:
- 20% or more. In this replay, the bias of all categories together stays between −12.8% and +10.2%. The four categories with no yearly season usually stay within ±15%. A limit of 20% catches the large misses, not the normal noise.
- Two weeks in a row, the same way. One bad week can be one event: a sunny weekend, a delivery problem. A bias that repeats is a pattern.
- 200 units or more. In a small category, a few units make a large percentage. Barbecues were −54.1% in the week of 24 March, but only 111 were sold. With 200 units as the floor, the alarm waits until the category matters.
The code is in forecast_job/monitor.py: check_week() writes check.csv, and alarms() applies the rule. The three numbers are at the top of the file: ALARM_PCT = 20.0, ALARM_WEEKS = 2 and MIN_UNITS = 200.
Check a week by hand
Section titled “Check a week by hand”-
In
forecast-job, with the virtual environment active, copy the test file of this lesson:Terminal window cp ../nextia-learning-labs/C35/forecast-job/end-of-m08/tests/test_monitor.py tests/Terminal window Copy-Item ..\nextia-learning-labs\C35\forecast-job\end-of-m08\tests\test_monitor.py tests\ -
Run the tests. The three new tests check that a normal week raises no alarm, that an early season does, and that one bad week does not.
Terminal window python -m pytest -q20 passed in 5.81s -
Check the plan of 24 March. The job planned it in the last lesson’s Your turn. If you skipped that, run
python -m forecast_job run --week 2025-03-24first.Terminal window python -m forecast_job check --week 2025-03-24Check 2025-03-24: all categories -7.5%, largest heating-and-decor +103.2%, 3 alarmsBIAS ALARM garden-furniture: the forecast was too low 2 weeks in a row (-45% and -59%, weeks of 2025-03-17, 2025-03-24). Look for the cause before the next order (a season that starts early, a stock-out, a price change), then refit or change the method.BIAS ALARM garden-tools: the forecast was too low 2 weeks in a row (-27% and -29%, weeks of 2025-03-17, 2025-03-24). Look for the cause before the next order (a season that starts early, a stock-out, a price change), then refit or change the method.BIAS ALARM heating-and-decor: the forecast was too high 2 weeks in a row (+81% and +103%, weeks of 2025-03-17, 2025-03-24). Look for the cause before the next order (a season that starts early, a stock-out, a price change), then refit or change the method.All categories together were only 7.5% too low. Inside that total, three categories were far off.
-
Open
runs/2025-03-24/check.csvin VS Code. It has one row per category:forecast,sales,bias_pctanddays(7, or fewer when the feed dropped a day).
If you check a week that has no plan, the job says so: The week of 2025-03-31 has no plan in runs/. Plan it first: …. If the week is not over yet, it says when to check it.
Replay the spring
Section titled “Replay the spring”The replay checks every week for you. Run the full quarter of 12 weeks. The weeks that you planned already stay as they are.
python -m forecast_job replay --start 2025-03-03 --weeks 12The output is long. These are its first lines, one week in the middle and the last line:
Week 2025-03-03 is already planned with the same inputs (ec8bedf4e7811156). Nothing to do.Week 2025-03-10 is already planned with the same inputs (244b2032b74c7443). Nothing to do.Check 2025-03-03: all categories +5.7%, largest plants-and-seeds -22.8%, 0 alarms…Check 2025-03-31: all categories -4.1%, largest heating-and-decor +72.3%, 3 alarms BIAS ALARM garden-furniture: the forecast was too low 2 weeks in a row (-59% and -41%, weeks of 2025-03-24, 2025-03-31). Look for the cause before the next order (a season that starts early, a stock-out, a price change), then refit or change the method. BIAS ALARM garden-tools: the forecast was too low 2 weeks in a row (-29% and -22%, weeks of 2025-03-24, 2025-03-31). Look for the cause before the next order (a season that starts early, a stock-out, a price change), then refit or change the method. BIAS ALARM heating-and-decor: the forecast was too high 2 weeks in a row (+103% and +72%, weeks of 2025-03-24, 2025-03-31). Look for the cause before the next order (a season that starts early, a stock-out, a price change), then refit or change the method.…Replayed 12 weeks from 2025-03-03: 17 bias alarms, 0 weeks stopped.The replay checked 11 weeks. The plan of 19 May is checked only when the job runs on 26 May, after the quarter. It raised 17 alarms in 6 of the 10 categories:
| Category | Alarms | Direction | When |
|---|---|---|---|
| heating-and-decor | 5 | too high | checks of 17 March to 14 April |
| garden-furniture | 4 | too low | 24 and 31 March, 5 and 12 May |
| plants-and-seeds | 2 | too low | 10 and 17 March |
| garden-tools | 2 | too low | 24 and 31 March |
| barbecue | 2 | too low | 5 and 12 May |
| shade-and-parasols | 2 | too low | 5 and 12 May |
The other four categories, bedding and textiles, kitchen, outdoor lighting and storage, raised no alarm in 11 weeks. This chart shows why:
- Heating and decor
- Garden furniture
- Kitchen
- The alarm limits, ±20%
Text description of the chart
The chart shows the bias of three categories over 11 weeks, with dashed lines at +20% and −20%. Heating and decor starts at +13.1%, and rises to +43.9%, +80.7% and +103.2% (week of 24 March). Then it is +72.3%, +32.0% and +36.5%. It drops to +7.5% on 21 April, and rises again to +62.3% on 5 May. Garden furniture starts near zero (−1.6%, −14.0%), then falls to −44.9%, −58.6% and −41.2%. It jumps to +15.0% on 7 April, and stays near −40% in late April and May. Kitchen stays between −5.0% and +21.0%, and only one week, 14 April, is above +20%.
The pattern is clear: the job is too high on heating while winter ends, and too low on everything for the garden while spring starts. Kitchen, bedding and storage have no season in the year, and the job forecasts them well.
Why the alarm fires: a season the method cannot see
Section titled “Why the alarm fires: a season the method cannot see”The alarm says: “look for the cause, then refit or change the method.” So Priya looks for the cause, with the four questions of this chart:
Text description of the diagram
The chart starts at a bias alarm: one category, two weeks off the same way. It asks: what is the cause? Four answers follow, each with its action. One: the data, for example a stock-out or a late or broken feed: fix the data, then run the week again. Two: a one-time event, for example a heat wave or a promotion: mark it and wait, the error goes away by itself. Three: a new level, for example a price rise that stays: refit, and the next weekly fit follows it. Four: a pattern the method cannot learn, for example a yearly season: change the method, add the yearly season or a driver.
Priya goes down the list:
- The data? No. The data check gave no warning in these weeks.
- A one-time event? No. The bias lasted for weeks, in six categories.
- A new level? No, or not only. A refit would fix that, and the job already refits every Monday. The alarms still fired week after week.
- A pattern the method cannot learn? Yes. The job’s ETS has a weekly season, but no yearly season. It learns the recent level and follows it with a delay. Every spring, garden products rise and heating falls, and the job is always a few weeks behind.
There is a second reason in 2025. The garden season came early. The 7-day average of garden furniture passed 150 units a day on 5 May 2025. In 2024, it did so on 7 June. That is about five weeks early (a situation we planted in the data). The warm spring made the delay worse, but the delay is there every spring.
So “refit” is the wrong answer here. The alarm says change the method. Which method?
- In March 2025, the job has less than a year of history (it starts on 1 April 2024). No method can learn a yearly season from less than one year. The best help is a person who knows the season: Kenji’s overrides, in the next lesson.
- From spring 2026, there are two years of history. Then a method with a yearly season can learn the spring ramp. One is MSTL with a weekly and a yearly season, from Several seasons and long series. Another is LightGBM with last year’s sales as a feature. Test the new method in the course backtest and in a replay before the job uses it.
The MLOps course asks the same question for any model: retrain on new data, or revise the model itself. A drift alarm that fires every month is a sign that retraining does not help.
Replay the autumn
Section titled “Replay the autumn”Is the alarm always on? Replay nine weeks of autumn:
python -m forecast_job replay --start 2025-09-29 --weeks 9The last lines:
Check 2025-11-17: all categories +3.5%, largest shade-and-parasols +42.7%, 1 alarms BIAS ALARM garden-tools: the forecast was too high 2 weeks in a row (+21% and +23%, weeks of 2025-11-10, 2025-11-17). Look for the cause before the next order (a season that starts early, a stock-out, a price change), then refit or change the method.Replayed 9 weeks from 2025-09-29: 4 bias alarms, 1 weeks stopped.The autumn has 4 alarms, against 17 in the spring, and one stopped week:
- The week of 20 October stopped: the missing day of 19 October. The job printed
The plan of the week before stays in use.The weeks after it planned normally, with the warningno sales rows for 2025-10-19: filled with the same weekday one week before and after. - Heating and decor, too low (−23% and −34%, weeks of 6 and 13 October). The heating season starts: sales rose from 247 to 843 units a week in seven weeks.
- Garden tools, too high, three times in November (about +21% to +23%). The garden season ends: sales fell from 1,463 to 930 units a week.
It is the same cause as in the spring, in the other direction. The alarm fires when a yearly season turns. In the weeks between, and in the categories with no yearly season, it stays quiet. In both replays, kitchen, bedding, storage and outdoor lighting raised no alarm in 18 checks.
When to use a bias alarm, and what it costs
Section titled “When to use a bias alarm, and what it costs”Use a bias alarm on every forecast that drives a decision. It is cheap: the job reads two small files per week. It needs only the plans that the store keeps.
It has limits and costs:
- It is late by design. It needs two finished weeks. Garden furniture was short in the weeks of 17 and 24 March before the first alarm came.
- Its numbers are choices. A lower limit raises more alarms, and people start to ignore them. A higher limit misses real problems. Choose them with the people who act on them, and test them in a replay.
- It checks one direction per category. It does not see a forecast that is right in total but wrong per product. For that, check the error at the level that Kenji orders, as in Hierarchies that must add up.
- Someone must act. An alarm that nobody reads is a log line. Name the person who reads it every Monday: here, Priya.
Common mistake
Refit more often
Symptom: after the first garden furniture alarm, Tomás says: “The model is old. Let us refit it every day.” The alarms continue on 31 March, then again in May.
Cause: the job already refits every Monday on the newest sales. A refit learns the recent level again; it does not add a pattern that the method does not have. ETS with only a weekly season will trail every yearly season, however often you fit it.
Fix: read the alarm as a question about the method. Here the cause is the yearly season. Until the history is long enough, add what a person knows (an override with its reason, next lesson). With two years of history, test a method with a yearly season in the backtest and in a replay, then change the job.
Consequence in AI work: “retrain more often” is the easy answer to every monitoring alarm. It costs compute and hides the real problem. When the same alarm fires after every retraining, the model itself must change.
Your turn: a stricter limit
Section titled “Your turn: a stricter limit”Grace asks: “Could the alarm fire less often, so that we take each one seriously?” Before you run the code, predict how many of the 17 spring alarms remain with a limit of 30% instead of 20%. Then run this in forecast-job (it reads the check.csv files of your replay, and changes no file):
python -c "import pandas as pdfrom forecast_job import monitorweeks = pd.date_range('2025-03-10', '2025-05-12', freq='7D')for limit in (20, 30): monitor.ALARM_PCT = limit found = [a for w in weeks for a in monitor.alarms('runs', w)] print(f'limit {limit}%: {len(found)} alarms')"Show the answer
limit 20%: 17 alarmslimit 30%: 14 alarmsOnly 3 alarms go away. Garden tools loses both of its alarms (−27%, −29% and −22% are under 30%). Plants and seeds loses its first one (−23% is under 30%). Heating and decor keeps all five: even its smallest bias, +32.0%, is above 30%. The other 14 alarms are 30% or more off, two weeks in a row. A stricter limit does not make this alarm quiet, because the misses are large. It fires on a real problem: the missing yearly season.
Knowledge check
Section titled “Knowledge check”Check your understanding
With the numbers of the garden furniture check, explain what the bias alarm measures and why the job raised it on 24 March.
Show the answer
The alarm compares the forecast for the first 7 days of a week’s plan with the units sold in those days. It does this per category: bias = (forecast − sales) ÷ sales. For garden furniture, the plan of 17 March forecast 111.9 units and 203 were sold (−44.9%). The plan of 24 March forecast 162.2 and 392 were sold (−58.6%). Both weeks were more than 20% off, in the same direction, with at least 200 units sold. So the job raised BIAS ALARM garden-furniture: the forecast was too low 2 weeks in a row.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The job refits every Monday, and the alarm keeps firing on garden products every spring. What does that tell you, and what would you change?
Show the answer
A refit learns the newest level again, but it does not change the method. The bias repeats in every category with a yearly season. It is too low on garden products in spring and too high on heating, and the other way round in autumn. Kitchen, bedding and storage raise no alarm. So the cause is a pattern that ETS with only a weekly season cannot learn. Change the method. With two years of history, use a method with a yearly season (MSTL, or LightGBM with last year’s sales). Test it in the backtest and in a replay first. With less than a year of history, add the planner’s knowledge as an override with its reason.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: in the week of 24 March, barbecues were −54.1% off with 111 units sold. The week before, −31.6% with 62 units. Is that an alarm? Should it be?
Show the answer
No alarm: both weeks were more than 20% off in the same direction, but fewer than 200 units were sold, so the rule waits. That is on purpose: in a small category, a few units make a large percentage, and an alarm on every small category would be ignored. The cost is a later alarm: barbecues raised one on 5 May, when sales passed 200 units a week. If barbecues matter more than their units, for example because they are expensive, the floor could be in dollars instead of units.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Every week, the job checks last week’s plan: bias = (forecast − sales) ÷ sales, per category, over the plan’s first 7 days.
- A bias alarm fires when a category is 20% or more off, in the same direction, two weeks in a row. Each week needs at least 200 units sold.
- In the spring replay, it fired 17 times, on garden products (too low) and heating (too high). In the autumn, 4 times, as the seasons turned the other way. Categories with no yearly season stayed quiet.
- After an alarm, find the cause. Fix the data, or wait out a one-time event. Refit for a new level. Change the method for a pattern that it cannot learn.
- Refit did not help here: the job refits every week. The alarm asks for a yearly season.
The job can now tell when its forecast is wrong. Kenji often knows it before the alarm does: he sees the warm weather coming. Next, you let him change the order, record his reason, and measure whether his changes helped.
Next: People and forecasts