Yesterday predicts today
- Lesson 5
- 30 min
- Module 2: Read a time series
Where we are
We can now see the trend and the seasons in a series. But we have not yet measured how much one day tells us about the next. In this lesson, we use lags and autocorrelation to measure the memory of a series, and we see why a random split breaks here.
Lesson 5 of 27 · Forecasting and Time Series: Predict What Comes NextYesterday predicts today
1. Forecasts serve decisions
2. Read a time series
3. Baselines and honest testing
4. Statistical forecasters
5. Machine learning forecasters
6. Hard series and many series
7. Uncertainty and the decision
8. Run a forecast in production
What you will be able to do
Measure how much a series remembers its past, with lags and the autocorrelation function. This memory is what a forecast uses. It is also why a random split gives a forecast an error that is too small.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It computes an autocorrelation by hand for 8 days. Then it computes the ACF of Larkfield’s orders, tickets and parasols, and runs the random split of the failure case. Its setup cells install pandas and statsforecast and download the practice data, so it needs no local setup. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The trend and the seasons of Larkfield’s orders, from Trend, season and noise.
- Train and test sets, and a random split against a time split. The machine learning course teaches them in Train, validation and test sets.
- The mean absolute error (MAE): the mean of the distances between the forecasts and the real values, from Regression metrics.
What you need on your computer
Nothing for the notebook route: it runs in Colab or Kaggle. For a local route, save the notebook in the folder larkfield that you made in Meet the Larkfield data, and choose the kernel .venv. The notebook reuses the data in larkfield-forecasting.
Tested with Python 3.12, pandas 2.3.3 and statsmodels 0.15.0 on macOS. Windows and Colab were not tested.
The weather tomorrow
Section titled “The weather tomorrow”What is the best simple guess for tomorrow’s temperature? For most places, it is “about the same as today”. A hot day is usually followed by another hot day. A cold week does not end in one night. The weather remembers its recent past.
It also remembers further back, but less. Today says a lot about tomorrow and a little about next week. The same day last year says something too: July is warm every year.
Daily orders behave in the same way. A busy Saturday is followed by a busy Sunday. A quiet January day sits between quiet January days. If we can measure this memory, we know which past days a forecast should look at.
The problem: “similar” is not a number
Section titled “The problem: “similar” is not a number”Priya wants to choose the inputs for her first forecasts. Should a forecast for Thursday look at Wednesday, at last Thursday, or at the same Thursday last year? “They look similar” is not enough. She needs a number for each choice, on the same scale, so that she can compare them.
There is a second problem. The machine learning course split rows into train and test sets at random for some models. If one day remembers the day before, then a random test day has its neighbours in the training data. Does that matter? We need the memory as a number to answer it.
The idea: lags and autocorrelation
Section titled “The idea: lags and autocorrelation”A lag is the series moved down
Section titled “A lag is the series moved down”A lag is the same series, moved down by a number of steps. The lag-1 column puts yesterday’s value next to today’s. The lag-7 column puts the value of one week ago next to today’s. In pandas, shift(1) and shift(7) make them:
lags = pd.DataFrame({"today": orders, "lag_1": orders.shift(1), "lag_7": orders.shift(7)}) today lag_1 lag_7date2024-04-01 550.0 NaN NaN2024-04-02 536.0 550.0 NaN2024-04-03 536.0 536.0 NaN2024-04-04 515.0 536.0 NaN2024-04-05 562.0 515.0 NaN2024-04-06 566.0 562.0 NaN2024-04-07 621.0 566.0 NaN2024-04-08 612.0 621.0 550.0The first row has no lag 1, because there is no earlier day. The lag-7 column starts on day 8. Every model in this course that uses the past uses lag columns like these.
Autocorrelation: does today follow yesterday?
Section titled “Autocorrelation: does today follow yesterday?”Correlation measures how two columns move together, from −1 to +1. At +1, when one column is above its average, the other is too, always. At 0, one tells you nothing about the other. At −1, they move in opposite directions.
Autocorrelation is the correlation of a series with its own lag. “Auto” means “self”: the series is compared with itself, moved by some steps. A high autocorrelation at lag 1 means that “above or below the average” today usually follows yesterday.
A tiny example: a bakery’s 8 days
Section titled “A tiny example: a bakery’s 8 days”A bakery sells loaves for 8 days. The mean is 26. For each day, we compute the distance from the mean. Then we multiply each day’s distance by yesterday’s distance.
| Day | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|---|---|---|---|---|---|---|---|---|
| Loaves | 20 | 23 | 25 | 24 | 28 | 30 | 27 | 31 |
| Distance from 26 | −6 | −3 | −1 | −2 | +2 | +4 | +1 | +5 |
| Yesterday’s distance | −6 | −3 | −1 | −2 | +2 | +4 | +1 | |
| Product | 18 | 3 | 2 | −4 | 8 | 4 | 5 |
A product is positive when today and yesterday are on the same side of the mean. Six of the seven products are positive. Only day 5 changes sides, so its product is negative.
The autocorrelation at lag 1 is the sum of the products divided by the sum of the squared distances:
- Sum of the products: 18 + 3 + 2 − 4 + 8 + 4 + 5 = 36.
- Sum of the squared distances: 36 + 9 + 1 + 4 + 4 + 16 + 1 + 25 = 96.
- Autocorrelation at lag 1: 36 ÷ 96 = 0.375.
The bottom number uses all 8 days, and the top number only 7 pairs, so a short series gives a smaller value. With 8 days, the number is also noisy. With 728 days, it becomes stable. statsmodels’ acf() does this computation, and it gives the same 0.375 for the bakery.
The autocorrelation function, or ACF, is this number for every lag: lag 1, lag 2, and so on. It is a profile of the series’ memory.
Larkfield’s memory
Section titled “Larkfield’s memory”Today against yesterday
Section titled “Today against yesterday”Each dot below is one day of Larkfield’s orders: yesterday’s value across, today’s value up.
Text description of the chart
A scatter plot with 727 dots and a straight line fitted through them, from about 100 to 1,400 orders on both axes. The dots form a rising band around a straight line. On the line, a day after 500 orders has about 530, and a day after 1,000 orders has about 945. A few dots sit far above the band, where today was much higher than yesterday. They are both Black Fridays, the heat-wave Saturday of 5 July 2025 and the Cyber Mondays. The days just after them sit far below the band.
The band rises: a high yesterday means a high today. The autocorrelation at lag 1 is 0.827. That is strong memory.
The ACF of the orders
Section titled “The ACF of the orders”Now the same number for every lag from 1 to 28 days:
- Lag 7, 14, 21 and 28 (same weekday)
- Other lags
Text description of the chart
One vertical bar for each lag from 1 to 28 days, on an axis from 0 to 1. Lag 1 is 0.827, lag 2 0.766, lag 3 0.722, lag 4 and lag 5 0.660, lag 6 0.662. Lag 7 rises again to 0.674, higher than lag 6. The same happens at lag 14 (0.571, after 0.561 at lag 13), lag 21 (0.479, after 0.475) and lag 28 (0.426, after 0.386). The bars fall slowly overall, from 0.827 to 0.426.
Read the chart in two steps.
- The bars fall slowly. Even 28 days back, the autocorrelation is 0.426. This slow fall comes from the yearly season: a summer day sits among summer days. Strong seasons and trends give slowly falling bars.
- Every seventh bar stands out. Lag 7 (0.674) is higher than lag 6 (0.662). Lags 14, 21 and 28 also beat their neighbours. The same weekday one, two, three and four weeks back is more like today than the days around it. That is the weekly season, measured.
So the best single day to look at is yesterday (0.827). The best day further back is the same weekday last week (0.674). Module 3 turns both into forecasts: “the same as yesterday” and “the same as last week”.
The ACF of the hourly tickets
Section titled “The ACF of the hourly tickets”Grace’s help desk works in hours. Here is the ACF of the hourly tickets, up to one week (168 hours):
Text description of the chart
A wave over 168 hours, on an axis from −0.6 to 1. It starts at 1 at lag 0 and 0.822 at lag 1. It falls below zero and reaches −0.438 at lag 12. It rises again to 0.810 at lag 24, one day. The same wave repeats every 24 hours, with peaks between 0.757 and 0.794. The highest peak after lag 0 is at lag 168, one week: 0.832.
The tickets have two seasons. The daily season makes the wave. The same hour yesterday is close (0.810). The hour twelve hours away is the opposite (−0.438): night against day. The weekly season makes the last peak the highest: the same hour last week (0.832) beats even the hour before (0.822). A Monday 10:00 is most like last Monday 10:00.
The parasol
Section titled “The parasol”In the notebook’s Your turn, you compute the parasol’s ACF. Its lag 1 is 0.827, the same as the orders. Its lag 7 is 0.675 and its lag 28 is 0.447, a little higher than the orders. The parasol’s memory comes mostly from its yearly season.
When the ACF helps, and what it cannot tell you
Section titled “When the ACF helps, and what it cannot tell you”Use the ACF before you build a forecast:
- It shows the season lengths to use: 7 for daily orders, 24 and 168 for hourly tickets. A method with the wrong season length misses the pattern.
- It shows which lags to give a machine learning model as inputs. Module 5 builds lag features from it.
- It shows how fast the memory fades. Slowly falling bars warn of a trend or a strong season. Modules 3 and 4 add methods for both.
The ACF has limits. It measures straight-line memory only. It mixes the effects: lag 2 is high partly because lag 1 is high, and yesterday also remembers the day before. And it says nothing about cause. Black Friday is not in the ACF at all, because it happens once a year on a moving date.
The ACF is cheap. It takes less than a second for two years of daily data. There is no reason to skip it.
Common mistake
A random split lets the neighbours tell the answer
Symptom: Tomás tests a simple rule: “forecast a day with the mean of the nearest days that we know, one before and one after”. He puts 20% of the days into a test set at random, as many machine learning projects do. The error is small, and he is pleased. Then Priya tests the same rule as a real forecast. At each of the course’s 13 cut-offs, from Meet the Larkfield data, she knows only the past. She forecasts the next 14 days. From the notebook’s section 5:
random split: 145 test days, MAE 59.3 ordersreal forecasts: 182 days, MAE 106.8 ordersText description of the chart
Two columns for the rule “the mean of the nearest known days”. On 145 random test days, its mean absolute error is 59.3 orders a day. On 182 days of real 14-day forecasts from 13 cut-offs, it is 106.8.
Cause: look at the first 14 days of the random split. “Test” marks the days that the random split chose for the test set.
| Day | 1 Apr | 2 Apr | 3 Apr | 4 Apr | 5 Apr | 6 Apr | 7 Apr | 8 Apr | 9 Apr | 10 Apr | 11 Apr | 12 Apr | 13 Apr | 14 Apr |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Set | train | test | train | test | train | test | train | train | train | test | train | train | train | test |
Every test day sits between two training days. The rule reads the day before and the day after. The ACF says that both are close to the answer (0.827 at lag 1). In real use, Kenji forecasts with the sales up to Sunday, for the next two weeks. He never knows the day after. The random split answers a question that nobody asks: “how well can we fill a gap in the past?”
Fix: test a forecast the way it is used. Choose a cut-off, train only on the days up to it, and score the days after it. Repeat this at several cut-offs. Test like the future: backtesting builds this test, and every method in this course uses it.
Consequence in forecasting: the random split here makes the error look 45% smaller. With more complex models, the harm is larger, because a model with lag features learns to copy the neighbours. A team that trusts the random score orders with a forecast that is far worse than it believes.
Guided practice: measure the memory in the notebook
Section titled “Guided practice: measure the memory in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell. The data cell ends with
Ready: 403758 rows of daily sales. - In section 1, predict the sign of the bakery’s autocorrelation. Then run the cell. You see
autocorrelation at lag 1: 0.375. - Run the
acf()cell. You see[1. 0.375]. - In section 2, run the lag cell. You see
NaNin the first row oflag_1and in the first 7 rows oflag_7. - In section 3, predict which lags stand out. Then run the ACF cell. You see
7: 0.674and28: 0.426. - Run the tickets cell. You see
24: 0.81and168: 0.832. - In section 5, predict which test gives the smaller error. Then run the cell. You see
MAE 59.3andMAE 106.8.
Your turn: the parasol’s memory
Section titled “Your turn: the parasol’s memory”Change one thing: the series. Compute the ACF of the parasol’s daily sales up to lag 28. Use load_sales(["P025"])["y"]. Before you run it, predict: will the parasol’s lag 7 be higher or lower than its lag 6? Put the value at lag 7, rounded to 3 decimals, in parasol_lag7.
Show the answer
parasol = load_sales(["P025"])["y"]acf_parasol = acf(parasol, nlags=28)parasol_lag7 = round(float(acf_parasol[7]), 3)print({k: round(float(acf_parasol[k]), 3) for k in (1, 7, 14, 28)}){1: 0.827, 7: 0.675, 14: 0.575, 28: 0.447}The parasol’s lag 7 is 0.675. Its memory falls more slowly than the orders’: 0.447 at lag 28, against 0.426. Its yearly season is stronger. A parasol day in July sits among July days, and a day in January among days with almost no sales.
Knowledge check
Section titled “Knowledge check”Check your understanding
In your own words, what does an autocorrelation of 0.827 at lag 1 say about Larkfield's daily orders? What does the higher value at lag 7 than at lag 6 add?
Show the answer
At lag 1, 0.827 means that the orders have strong memory. When yesterday was above the average, today is usually above it too, and the same for below. Yesterday is the best single day to look at. Lag 7 (0.674) is higher than lag 6 (0.662), and lags 14, 21 and 28 also beat their neighbours. The same weekday in earlier weeks is more like today than the days around it: this is the weekly season. A forecast should look at yesterday and at the same weekday last week.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: Grace wants to forecast the tickets for next Monday at 10:00. Which single past hour is the best guide, by the ACF of the hourly tickets, and which hour is the worst?
Show the answer
The best is the same hour one week before, last Monday at 10:00 (lag 168, 0.832). It is even better than the hour just before (0.822), and Grace does not know that hour a week ahead anyway. The same hour on the day before is also good (lag 24, 0.810). The worst is an hour twelve hours away, such as 22:00 (lag 12, −0.438): night and day move in opposite directions.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás says: 'My random split gives an error of 59 orders a day, so the rule is good enough.' Explain to him why this number is too good, and how to get an honest one.
Show the answer
The orders have strong memory: neighbouring days are close (0.827 at lag 1). In a random split, almost every test day has its neighbours, the day before and the day after, in the training data. The rule copies the answer from them. In real use, the days after the cut-off are never known. Tested as real 14-day forecasts from 13 cut-offs, the same rule has an error of 106.8 orders a day. For an honest number, train only on the days up to a cut-off and score the days after it. Repeat this at several cut-offs.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A lag is the series moved down by some steps:
shift(1)puts yesterday next to today. - Autocorrelation is the correlation of a series with its own lag, from −1 to +1. The ACF shows it for every lag: a profile of the series’ memory.
- Larkfield’s orders: 0.827 at lag 1, and the same weekday stands out (0.674 at lag 7). Hourly tickets peak every 24 hours (0.810) and highest at one week (0.832).
- The ACF shows the season lengths and the useful lags. It does not show moving holidays or causes.
- Strong memory breaks a random split: the neighbours give away the answer. One rule had an error of 59.3 orders a day with a random split and 106.8 as real forecasts.
You can now measure how a series remembers its past. Some days, the past is not the best guide: Easter moves, and a promotion doubles sales. Next, you put these events in a calendar.