Skip to content
Français

Clean without hiding demand

  • Lesson 7
  • 35 min
  • Module 2: Read a time series

Where we are

We know the events that move demand. The data still has holes: a missing day, a stock-out that looks like no demand, a one-off spike. In this lesson, we decide for each one whether to fill it, mark it or model it, without hiding real demand.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Clean the holes and odd values in a series so that the forecast learns real demand. A stock-out recorded as zero sales teaches a forecast that customers stopped buying, and the next order is far too small.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It finds the stock-outs and trains a forecast on a stock-out two ways. Then it fills the missing day, looks at the heat-wave spike and compares three products. Its setup cells install pandas and statsforecast and download the practice data, so it needs no local setup. About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

Nothing for the notebook route: it runs in Colab or Kaggle. For a local route, save the notebook in the folder larkfield that you made in Meet the Larkfield data, and choose the kernel .venv. The notebook reuses the data in larkfield-forecasting.

Tested with Python 3.12, pandas 2.3.3 and statsforecast 2.1.1 on macOS. Windows and Colab were not tested.

A bakery bakes 30 loaves a day. On a busy Saturday, the bread is gone at 11:00. Customers come all afternoon, see the empty shelf and leave. The till records 30 loaves sold.

Next week, the baker plans her baking from the till. Saturday says 30, so she bakes 30 again, and the shelf is empty again. The till never shows the customers who left. Her plan repeats her mistake, week after week.

The till records sales: what customers bought. The baker needs demand: what customers wanted. Most days they are the same. On a day with an empty shelf, sales are lower than demand, and the difference is invisible.

Larkfield’s data has three kinds of odd values, and each looks different in the file.

  1. A missing day. On 19 October 2025, the sales and orders feed dropped a day. There are no rows for it at all, a situation we planted. The ticket files are complete.
  2. Stock-outs. On some days, a product’s shelf in a region was empty. The column stock_out is 1 on those days. There are 2,473 such rows, 0.6% of the data, in 289 of the 600 product-region series.
  3. Spikes. Some days are far above normal: Black Friday, a promotion week, the heat wave of July 2025.

None of these raises an error. A forecaster who reads the file as it is learns from all three. Here is the worst case.

The bakery has 30 loaves a day. For five days, it also counted the customers it sent away, so we know the demand:

Day 1 2 3 4 5 Mean
Demand 24 27 41 38 25 31.0
Sales (at most 30) 24 27 30 30 25 27.2
stock_out 0 0 1 1 0

On days 3 and 4, the shelf was empty, and the sales hide 11 + 8 = 19 loaves. The mean of the sales is 27.2, but customers wanted 31.0 a day. A forecast trained on the sales plans 27, and the shelf is empty more often. The bias is built in: sales can only be lower than demand, never higher.

Larkfield’s stock-out: P002 in the north

Section titled “Larkfield’s stock-out: P002 in the north”

P002 is a black folding chair. In the north region, its shelf was empty for nine days, from 12 to 20 May 2025, at the start of the garden season. You met this stock-out in Frame the forecast; here we look at it closely. It is a situation we planted. The sales were 0 on all nine days. The hidden truth, which the course keeps back and you never see in the data, says that customers wanted 48 chairs in those days.

On 20 May, Kenji orders chairs for the next 14 days. Priya trains the same forecaster two ways on the history up to that day. The forecaster is statsforecast’s AutoETS, a method of Exponential smoothing. Here, read it as “a forecaster that learns the recent level”.

  • Zeros: the sales as they are, with the nine stock-out days as 0.
  • Marked: the stock-out days marked as missing, then filled with the mean of the same weekday one and two weeks before.
  • Units sold per day
  • Stock-out days
  • Forecast, trained on the zeros
  • Forecast, days marked and filled
Nine stock-out days with zero sales, then two 14-day forecasts. Trained on the zeros, the forecast is 0.5 chairs a day; with the days marked, 2.0. The chairs really sold 52 in the 14 days.
Text description of the chart

Daily sales of P002 in the north from 14 April to 3 June 2025, mostly between 0 and 5 chairs a day. From 12 to 20 May, nine days are marked as stock-out days, all with 0 sales. From 21 May, two flat forecast lines start. The forecast trained on the zeros is 0.53 chairs a day (7 in 14 days). The forecast with the days marked and filled is 2.0 a day (28 in 14 days). The real sales in these 14 days are between 0 and 7 a day, 52 in total. Seven of the days are a promotion week, 26 May to 1 June.

The 14-day forecast trained on the zeros is 7 chairs (bias −85.6%); with the stock-out marked it is 28 (bias −46.3%). The real sales were 52.
Text description of the chart

Three columns for the 14 days from 21 May to 3 June 2025. The forecast trained on the zeros is 7 chairs. The forecast with the stock-out days marked and filled is 28. The real sales are 52.

Trained on the zeros, the forecaster learns that the chair stopped selling. Its recent level is near 0, so it forecasts 0.53 chairs a day, 7 in 14 days. That is a bias of −85.6%. Kenji would order 7 chairs, and the shelf would be empty again, as for the baker.

With the days marked, the forecast is 28 chairs, a bias of −46.3%. It is four times closer, but it is still too low. Two reasons are visible in the data. The filled values come from the two weeks before the stock-out, when the season had not yet started. They average 1.8 chairs a day. But the hidden demand in the stock-out was 48 chairs in nine days, 5.3 a day. And a promotion week, 26 May to 1 June, falls inside the 14 days, which this forecaster does not know about. Calendars, holidays and events showed the fix for that part.

Marking does not invent the missing demand. It stops the forecast from learning a false fact: “nobody wants this chair”.

Each kind of odd value needs its own treatment. The question is always: was this real demand?

A decision diagram. An odd value in the history leads to four cases. No value at all, because the feed dropped a day: fill it with the mean of the same weekday one week before and after. Sales lower than demand, a stock-out: mark the days, then fill them; treat them as missing, never as zero demand. Real demand that comes back, a promotion or a holiday: model it with a feature from the calendar or the plan. Real demand one time only, a heat wave: mark it and keep it, and check its weight in the next forecast.
Four kinds of odd value and what to do with each. Only the first two are wrong data; the last two are real demand.
Text description of the diagram

The diagram starts from one box, “an odd value in the history”, and splits into four cases. A missing day, with no value at all, is filled with the mean of the same weekday one week before and one week after. Sales lower than demand, a stock-out, are marked and then filled: the days are treated as missing, never as zero demand. Real demand that comes back, such as a promotion or a holiday, is modelled with a feature from the calendar or the promotion plan. Real demand that happens one time only, such as the heat wave, is marked and kept, and its weight in the next forecast is checked.

A missing day is a hole in the record, not a fact about customers. Fill it so that the series has one value every day. This course fills it with the mean of the same weekday one week before and one week after. The ACF showed that the same weekday is the best guide further back.

Sun 12 Oct Sun 19 Oct Sun 26 Oct
Orders 695 missing 592
Filled (695 + 592) ÷ 2 = 643.5, rounded to 644
The orders around the missing day. The filled value, 643.5, sits among the other Sundays.
Text description of the chart

Daily orders from 5 October to 1 November 2025, between 518 and 710. Sunday 19 October has no data and is filled with 643.5, the mean of Sunday 12 October (695) and Sunday 26 October (592). The filled point sits among the other days of the series.

Find the hole first. In pandas, asfreq("D") gives the series one row per day, so the missing day becomes an empty value (NaN). Without it, the series jumps from 18 to 20 October. Then every “7 days before” after the gap points at the wrong weekday, and the error is silent.

A stock-out is not a hole: the shelf really sold 0. But 0 is not the demand. Mark the days with the flag, then treat their sales as missing and fill them as above, from the same weekday. Some methods can also leave them out. Never let them count as zero demand.

A promotion week or Black Friday is real demand. Do not delete it, and do not smooth it away. It will come back, on a date you know. Give the forecast a column for it, as in the last lesson, so it learns the effect apart from the normal level.

Keep: a real spike that will not come back

Section titled “Keep: a real spike that will not come back”

The heat-wave week of 30 June to 6 July 2025 was real. The parasol sold 147 that week, against 50 the week before. It will not come back on a known date. Keep it, because customers really bought those parasols. But mark it, and check how much the next forecast follows it. A forecast that copies last week would order 147 parasols for an ordinary week.

Finding unusual values like this one in hundreds of series is the subject of the anomaly detection course. That course finds the points; this course decides what a forecast should learn from them.

Clean every series before you train and before you score. Score on the cleaned days too: a forecast should not be blamed for a stock-out day where the customers could not buy.

Cleaning has costs and limits:

  • Filled values are guesses. The P002 fill was too low, because the season was starting. Keep the flag, so a later method can treat the days differently.
  • A flag depends on the data feed. Mei must make sure that stock_out is recorded. A stock-out without a flag looks exactly like low demand.
  • Do not clean real demand. Deleting every spike makes the forecast too smooth, and the shop runs short every Black Friday.
  • Write it down. Record every rule (which days, which fill) so that the forecasting job in Module 8 applies the same cleaning every week.

Common mistake

A stock-out trained as zero demand

Symptom: a product’s forecast falls to almost nothing right after a period of zero sales, while nothing else changed. For P002 in the north, the 14-day forecast after the nine stock-out days is 7 chairs. The product really sold 52:

zeros 14-day forecast 7.5, real sales 52, bias -85.6%
marked 14-day forecast 27.9, real sales 52, bias -46.3%

Cause: the forecaster learned from sales, not demand. On the nine stock-out days, the sales were 0 because the shelf was empty. The forecaster cannot know that, so it learned a new, low level.

Diagnosis: look for zero runs inside a series that normally sells, and check the stock_out column on those days. Plot the series with the stock-out days marked, as in the chart above.

Fix: mark the stock-out days as missing and fill them from the same weekday one and two weeks before. Then train again. The forecast rises from 7 to 28 chairs, four times closer to the real 52.

Consequence in forecasting: the error feeds itself. A low forecast means a small order, so the shelf is empty again, the sales are 0 again, and the next forecast is even lower. The data then “proves” that nobody wants the product. Kenji loses sales, and the chair may even be removed from the range.

Guided practice: clean a series in the notebook

Section titled “Guided practice: clean a series in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the helper cell. The data cell ends with Ready: 403758 rows of daily sales.
  3. In section 1, predict the difference of the two means. Then run the cell. You see mean demand 31.0 | mean sales 27.2.
  4. In section 2, run the cell. You see 2473 stock-out rows and nine days of stock_out 1 for P002.
  5. In section 3, predict which forecast is closer. Then run the cell. You see 7.5 and 27.9, and real sales 52.
  6. In section 4, run the cell. You see rows on 2025-10-19: 0 and filled: 644.
  7. In section 5, run the cell. You see 147 for the week of 6 July.

This task uses the whole module. Take three products: P025 (parasol), P077 (garden hose) and P141 (duvet). Build a table summary with their stock-out days, their promotion days and the remainder’s share of the variance from MSTL. Then say which product will be the hardest to forecast, and why. Before you run it, predict which product has the largest remainder share.

Show the answer
rows = {}
for pid in ["P025", "P077", "P141"]:
s = load_sales([pid])
sf = StatsForecast(models=[MSTL(season_length=[7, 365])], freq="D")
sf.fit(s[["unique_id", "ds", "y"]])
parts = sf.fitted_[0, 0].model_
rows[pid] = {"stock_out_days": int(s["stock_out"].sum()), "promotion_days": int(s["on_promotion"].sum()),
"remainder_share": round(float(100 * parts["remainder"].var() / parts["data"].var()), 1)}
summary = pd.DataFrame.from_dict(rows, orient="index")
print(summary)
stock_out_days promotion_days remainder_share
P025 9 14 19.1
P077 0 14 27.7
P141 13 7 32.7

P141 is the hardest: the largest remainder (32.7%) and the most stock-out days (13). Its promotion doubles the sales, and a season cannot explain it, so it stays in the remainder; its stock-outs hide demand. A promotion column and marked stock-outs fix much of it. P077 has no stock-outs, but its price rise of June 2025 moves the level for good. The parasol has the strongest season and a stock-out in the heat wave. Each product is hard in its own way, and the parts tell you why.

Check your understanding

Show one stock-out from this lesson and explain why training on it as zero demand gives a forecast that is too low.

Show the answer

P002, a folding chair, had an empty shelf in the north from 12 to 20 May 2025. The sales were 0 on all nine days, but customers wanted 48 chairs (the hidden truth). A forecaster trained on the sales sees nine days of 0 and learns that the chair stopped selling. Its 14-day forecast is 7 chairs, while the chair really sold 52: a bias of −85.6%. Sales can only be lower than demand, never higher, so the error always goes the same way. With the days marked as missing and filled, the forecast is 28.

Check your understanding

For each case, say whether you fill it, mark it, model it or keep it, and why: (a) no rows for 19 October 2025; (b) Black Friday; (c) the parasol's heat-wave week; (d) nine days of zero sales with stock_out = 1.

Show the answer

(a) Fill: it is a hole in the record, not a fact about customers. Use the mean of the same weekday one week before and after (643.5 orders). (b) Model: it is real demand that comes back on a known date, so give the forecast a Black Friday column. (c) Keep and mark: it was real demand, but it will not come back on a known date. Check how much the next forecast follows it. (d) Mark, then fill: the sales are not the demand, so treat the days as missing, never as zero.

Check your understanding

Predict: Tomás deletes every day that is more than three times the series' median, to 'remove the outliers'. What happens to the forecast for next Black Friday and for the next promotion week?

Show the answer

Both forecasts are too low. Black Friday and the promotion weeks are real demand that comes back. Black Friday has twice the orders of a normal Friday or more, and a median promotion ×1.87. Deleting them removes exactly the days that teach the forecast these effects. The forecast becomes too smooth, and the shop runs short on the busiest days. Keep real demand. Give the events their own columns, and fill only the days where the data is wrong or the shelf was empty.

Recap

  • Sales are not demand. On a stock-out day, the sales are lower than what customers wanted. Larkfield has 2,473 stock-out rows (0.6%), flagged in stock_out.
  • P002’s nine-day stock-out, trained as zeros, gives a 14-day forecast of 7 chairs (bias −85.6%). With the days marked and filled, it is 28 (−46.3%). The chair sold 52.
  • Fill a missing day (19 October 2025: 643.5 orders, from the same weekday before and after). Mark a stock-out, then fill it. Model events that come back. Keep and mark a real one-off spike.
  • Find the holes first: asfreq("D") makes a missing day visible.
  • In this module, you learned to read a series: its parts, its memory, its events and its holes.

You can now read a series and clean it without hiding demand. But we have not yet made a single forecast. Next, you build the simplest forecasts there are, and you see how hard they are to beat.

Next: Naive forecasts