Clean without hiding demand
- Lesson 7
- 35 min
- Module 2: Read a time series
Where we are
We know the events that move demand. The data still has holes: a missing day, a stock-out that looks like no demand, a one-off spike. In this lesson, we decide for each one whether to fill it, mark it or model it, without hiding real demand.
Lesson 7 of 27 · Forecasting and Time Series: Predict What Comes NextClean without hiding demand
1. Forecasts serve decisions
2. Read a time series
3. Baselines and honest testing
4. Statistical forecasters
5. Machine learning forecasters
6. Hard series and many series
7. Uncertainty and the decision
8. Run a forecast in production
What you will be able to do
Clean the holes and odd values in a series so that the forecast learns real demand. A stock-out recorded as zero sales teaches a forecast that customers stopped buying, and the next order is far too small.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It finds the stock-outs and trains a forecast on a stock-out two ways. Then it fills the missing day, looks at the heat-wave spike and compares three products. Its setup cells install pandas and statsforecast and download the practice data, so it needs no local setup. About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The decomposition with MSTL, from Trend, season and noise.
- The weekly season in the ACF, from Yesterday predicts today: the same weekday is the best guide further back.
- Promotions as known future inputs, from Calendars, holidays and events.
- Bias: the forecast total minus the real total, as a share of the real total. A negative bias means the forecast is too low. Measure the error that matters uses it in every module.
What you need on your computer
Nothing for the notebook route: it runs in Colab or Kaggle. For a local route, save the notebook in the folder larkfield that you made in Meet the Larkfield data, and choose the kernel .venv. The notebook reuses the data in larkfield-forecasting.
Tested with Python 3.12, pandas 2.3.3 and statsforecast 2.1.1 on macOS. Windows and Colab were not tested.
The empty shelf at the bakery
Section titled “The empty shelf at the bakery”A bakery bakes 30 loaves a day. On a busy Saturday, the bread is gone at 11:00. Customers come all afternoon, see the empty shelf and leave. The till records 30 loaves sold.
Next week, the baker plans her baking from the till. Saturday says 30, so she bakes 30 again, and the shelf is empty again. The till never shows the customers who left. Her plan repeats her mistake, week after week.
The till records sales: what customers bought. The baker needs demand: what customers wanted. Most days they are the same. On a day with an empty shelf, sales are lower than demand, and the difference is invisible.
The problem: holes that look like data
Section titled “The problem: holes that look like data”Larkfield’s data has three kinds of odd values, and each looks different in the file.
- A missing day. On 19 October 2025, the sales and orders feed dropped a day. There are no rows for it at all, a situation we planted. The ticket files are complete.
- Stock-outs. On some days, a product’s shelf in a region was empty. The column
stock_outis 1 on those days. There are 2,473 such rows, 0.6% of the data, in 289 of the 600 product-region series. - Spikes. Some days are far above normal: Black Friday, a promotion week, the heat wave of July 2025.
None of these raises an error. A forecaster who reads the file as it is learns from all three. Here is the worst case.
A tiny example: what the till hides
Section titled “A tiny example: what the till hides”The bakery has 30 loaves a day. For five days, it also counted the customers it sent away, so we know the demand:
| Day | 1 | 2 | 3 | 4 | 5 | Mean |
|---|---|---|---|---|---|---|
| Demand | 24 | 27 | 41 | 38 | 25 | 31.0 |
| Sales (at most 30) | 24 | 27 | 30 | 30 | 25 | 27.2 |
stock_out |
0 | 0 | 1 | 1 | 0 |
On days 3 and 4, the shelf was empty, and the sales hide 11 + 8 = 19 loaves. The mean of the sales is 27.2, but customers wanted 31.0 a day. A forecast trained on the sales plans 27, and the shelf is empty more often. The bias is built in: sales can only be lower than demand, never higher.
Larkfield’s stock-out: P002 in the north
Section titled “Larkfield’s stock-out: P002 in the north”P002 is a black folding chair. In the north region, its shelf was empty for nine days, from 12 to 20 May 2025, at the start of the garden season. You met this stock-out in Frame the forecast; here we look at it closely. It is a situation we planted. The sales were 0 on all nine days. The hidden truth, which the course keeps back and you never see in the data, says that customers wanted 48 chairs in those days.
On 20 May, Kenji orders chairs for the next 14 days. Priya trains the same forecaster two ways on the history up to that day. The forecaster is statsforecast’s AutoETS, a method of Exponential smoothing. Here, read it as “a forecaster that learns the recent level”.
- Zeros: the sales as they are, with the nine stock-out days as 0.
- Marked: the stock-out days marked as missing, then filled with the mean of the same weekday one and two weeks before.
- Units sold per day
- Stock-out days
- Forecast, trained on the zeros
- Forecast, days marked and filled
Text description of the chart
Daily sales of P002 in the north from 14 April to 3 June 2025, mostly between 0 and 5 chairs a day. From 12 to 20 May, nine days are marked as stock-out days, all with 0 sales. From 21 May, two flat forecast lines start. The forecast trained on the zeros is 0.53 chairs a day (7 in 14 days). The forecast with the days marked and filled is 2.0 a day (28 in 14 days). The real sales in these 14 days are between 0 and 7 a day, 52 in total. Seven of the days are a promotion week, 26 May to 1 June.
Text description of the chart
Three columns for the 14 days from 21 May to 3 June 2025. The forecast trained on the zeros is 7 chairs. The forecast with the stock-out days marked and filled is 28. The real sales are 52.
Trained on the zeros, the forecaster learns that the chair stopped selling. Its recent level is near 0, so it forecasts 0.53 chairs a day, 7 in 14 days. That is a bias of −85.6%. Kenji would order 7 chairs, and the shelf would be empty again, as for the baker.
With the days marked, the forecast is 28 chairs, a bias of −46.3%. It is four times closer, but it is still too low. Two reasons are visible in the data. The filled values come from the two weeks before the stock-out, when the season had not yet started. They average 1.8 chairs a day. But the hidden demand in the stock-out was 48 chairs in nine days, 5.3 a day. And a promotion week, 26 May to 1 June, falls inside the 14 days, which this forecaster does not know about. Calendars, holidays and events showed the fix for that part.
Marking does not invent the missing demand. It stops the forecast from learning a false fact: “nobody wants this chair”.
The idea: fill, mark or model
Section titled “The idea: fill, mark or model”Each kind of odd value needs its own treatment. The question is always: was this real demand?
Text description of the diagram
The diagram starts from one box, “an odd value in the history”, and splits into four cases. A missing day, with no value at all, is filled with the mean of the same weekday one week before and one week after. Sales lower than demand, a stock-out, are marked and then filled: the days are treated as missing, never as zero demand. Real demand that comes back, such as a promotion or a holiday, is modelled with a feature from the calendar or the promotion plan. Real demand that happens one time only, such as the heat wave, is marked and kept, and its weight in the next forecast is checked.
Fill: the missing day
Section titled “Fill: the missing day”A missing day is a hole in the record, not a fact about customers. Fill it so that the series has one value every day. This course fills it with the mean of the same weekday one week before and one week after. The ACF showed that the same weekday is the best guide further back.
| Sun 12 Oct | Sun 19 Oct | Sun 26 Oct | |
|---|---|---|---|
| Orders | 695 | missing | 592 |
| Filled | (695 + 592) ÷ 2 = 643.5, rounded to 644 |
Text description of the chart
Daily orders from 5 October to 1 November 2025, between 518 and 710. Sunday 19 October has no data and is filled with 643.5, the mean of Sunday 12 October (695) and Sunday 26 October (592). The filled point sits among the other days of the series.
Find the hole first. In pandas, asfreq("D") gives the series one row per day, so the missing day becomes an empty value (NaN). Without it, the series jumps from 18 to 20 October. Then every “7 days before” after the gap points at the wrong weekday, and the error is silent.
Mark: the stock-outs
Section titled “Mark: the stock-outs”A stock-out is not a hole: the shelf really sold 0. But 0 is not the demand. Mark the days with the flag, then treat their sales as missing and fill them as above, from the same weekday. Some methods can also leave them out. Never let them count as zero demand.
Model: events that come back
Section titled “Model: events that come back”A promotion week or Black Friday is real demand. Do not delete it, and do not smooth it away. It will come back, on a date you know. Give the forecast a column for it, as in the last lesson, so it learns the effect apart from the normal level.
Keep: a real spike that will not come back
Section titled “Keep: a real spike that will not come back”The heat-wave week of 30 June to 6 July 2025 was real. The parasol sold 147 that week, against 50 the week before. It will not come back on a known date. Keep it, because customers really bought those parasols. But mark it, and check how much the next forecast follows it. A forecast that copies last week would order 147 parasols for an ordinary week.
Finding unusual values like this one in hundreds of series is the subject of the anomaly detection course. That course finds the points; this course decides what a forecast should learn from them.
When to clean, and what it costs
Section titled “When to clean, and what it costs”Clean every series before you train and before you score. Score on the cleaned days too: a forecast should not be blamed for a stock-out day where the customers could not buy.
Cleaning has costs and limits:
- Filled values are guesses. The P002 fill was too low, because the season was starting. Keep the flag, so a later method can treat the days differently.
- A flag depends on the data feed. Mei must make sure that
stock_outis recorded. A stock-out without a flag looks exactly like low demand. - Do not clean real demand. Deleting every spike makes the forecast too smooth, and the shop runs short every Black Friday.
- Write it down. Record every rule (which days, which fill) so that the forecasting job in Module 8 applies the same cleaning every week.
Common mistake
A stock-out trained as zero demand
Symptom: a product’s forecast falls to almost nothing right after a period of zero sales, while nothing else changed. For P002 in the north, the 14-day forecast after the nine stock-out days is 7 chairs. The product really sold 52:
zeros 14-day forecast 7.5, real sales 52, bias -85.6%marked 14-day forecast 27.9, real sales 52, bias -46.3%Cause: the forecaster learned from sales, not demand. On the nine stock-out days, the sales were 0 because the shelf was empty. The forecaster cannot know that, so it learned a new, low level.
Diagnosis: look for zero runs inside a series that normally sells, and check the stock_out column on those days. Plot the series with the stock-out days marked, as in the chart above.
Fix: mark the stock-out days as missing and fill them from the same weekday one and two weeks before. Then train again. The forecast rises from 7 to 28 chairs, four times closer to the real 52.
Consequence in forecasting: the error feeds itself. A low forecast means a small order, so the shelf is empty again, the sales are 0 again, and the next forecast is even lower. The data then “proves” that nobody wants the product. Kenji loses sales, and the chair may even be removed from the range.
Guided practice: clean a series in the notebook
Section titled “Guided practice: clean a series in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell. The data cell ends with
Ready: 403758 rows of daily sales. - In section 1, predict the difference of the two means. Then run the cell. You see
mean demand 31.0 | mean sales 27.2. - In section 2, run the cell. You see
2473 stock-out rowsand nine days ofstock_out1 for P002. - In section 3, predict which forecast is closer. Then run the cell. You see
7.5and27.9, andreal sales 52. - In section 4, run the cell. You see
rows on 2025-10-19: 0andfilled: 644. - In section 5, run the cell. You see
147for the week of 6 July.
Your turn: which product is the hardest?
Section titled “Your turn: which product is the hardest?”This task uses the whole module. Take three products: P025 (parasol), P077 (garden hose) and P141 (duvet). Build a table summary with their stock-out days, their promotion days and the remainder’s share of the variance from MSTL. Then say which product will be the hardest to forecast, and why. Before you run it, predict which product has the largest remainder share.
Show the answer
rows = {}for pid in ["P025", "P077", "P141"]: s = load_sales([pid]) sf = StatsForecast(models=[MSTL(season_length=[7, 365])], freq="D") sf.fit(s[["unique_id", "ds", "y"]]) parts = sf.fitted_[0, 0].model_ rows[pid] = {"stock_out_days": int(s["stock_out"].sum()), "promotion_days": int(s["on_promotion"].sum()), "remainder_share": round(float(100 * parts["remainder"].var() / parts["data"].var()), 1)}summary = pd.DataFrame.from_dict(rows, orient="index")print(summary) stock_out_days promotion_days remainder_shareP025 9 14 19.1P077 0 14 27.7P141 13 7 32.7P141 is the hardest: the largest remainder (32.7%) and the most stock-out days (13). Its promotion doubles the sales, and a season cannot explain it, so it stays in the remainder; its stock-outs hide demand. A promotion column and marked stock-outs fix much of it. P077 has no stock-outs, but its price rise of June 2025 moves the level for good. The parasol has the strongest season and a stock-out in the heat wave. Each product is hard in its own way, and the parts tell you why.
Knowledge check
Section titled “Knowledge check”Check your understanding
Show one stock-out from this lesson and explain why training on it as zero demand gives a forecast that is too low.
Show the answer
P002, a folding chair, had an empty shelf in the north from 12 to 20 May 2025. The sales were 0 on all nine days, but customers wanted 48 chairs (the hidden truth). A forecaster trained on the sales sees nine days of 0 and learns that the chair stopped selling. Its 14-day forecast is 7 chairs, while the chair really sold 52: a bias of −85.6%. Sales can only be lower than demand, never higher, so the error always goes the same way. With the days marked as missing and filled, the forecast is 28.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
For each case, say whether you fill it, mark it, model it or keep it, and why: (a) no rows for 19 October 2025; (b) Black Friday; (c) the parasol's heat-wave week; (d) nine days of zero sales with stock_out = 1.
Show the answer
(a) Fill: it is a hole in the record, not a fact about customers. Use the mean of the same weekday one week before and after (643.5 orders). (b) Model: it is real demand that comes back on a known date, so give the forecast a Black Friday column. (c) Keep and mark: it was real demand, but it will not come back on a known date. Check how much the next forecast follows it. (d) Mark, then fill: the sales are not the demand, so treat the days as missing, never as zero.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: Tomás deletes every day that is more than three times the series' median, to 'remove the outliers'. What happens to the forecast for next Black Friday and for the next promotion week?
Show the answer
Both forecasts are too low. Black Friday and the promotion weeks are real demand that comes back. Black Friday has twice the orders of a normal Friday or more, and a median promotion ×1.87. Deleting them removes exactly the days that teach the forecast these effects. The forecast becomes too smooth, and the shop runs short on the busiest days. Keep real demand. Give the events their own columns, and fill only the days where the data is wrong or the shelf was empty.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Sales are not demand. On a stock-out day, the sales are lower than what customers wanted. Larkfield has 2,473 stock-out rows (0.6%), flagged in
stock_out. - P002’s nine-day stock-out, trained as zeros, gives a 14-day forecast of 7 chairs (bias −85.6%). With the days marked and filled, it is 28 (−46.3%). The chair sold 52.
- Fill a missing day (19 October 2025: 643.5 orders, from the same weekday before and after). Mark a stock-out, then fill it. Model events that come back. Keep and mark a real one-off spike.
- Find the holes first:
asfreq("D")makes a missing day visible. - In this module, you learned to read a series: its parts, its memory, its events and its holes.
You can now read a series and clean it without hiding demand. But we have not yet made a single forecast. Next, you build the simplest forecasts there are, and you see how hard they are to beat.
Next: Naive forecasts