Skip to content
Français

Gradient boosting with LightGBM

  • Lesson 15
  • 45 min
  • Module 5: Machine learning forecasters

Where we are

In the last lesson, we built forecasting features by hand and fit a linear regression. It used the promotion flag, but it added the same number of units for every product. In this lesson, we train one LightGBM model for all products with mlforecast, forecast many days ahead in two ways, and avoid the leakage traps that rolling features hide. Think first of a golfer on the green.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Train one gradient boosting model for all products, forecast 14 days ahead without breaking the gap, and recognise a rolling feature that leaks the future. LightGBM gives the lowest order cost of the course table, if you set it up honestly.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It trains LightGBM, backtests three set-ups and finds the leak in a rolling mean. Its setup cells install the packages and download the Larkfield data, so it needs no local setup. About 45 minutes, of which 2 to 3 minutes of waiting.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

What you need on your computer

Nothing: the notebook runs in Colab or Kaggle. Its first cell installs mlforecast 1.1.0 and LightGBM 4.7.0.

Tested with Python 3.12, pandas 2.3.3, mlforecast 1.1.0 and LightGBM 4.7.0, in Jupyter on macOS.

A golfer is on the green, 10 metres from the hole. The first putt stops 2 metres short. The second putt aims only at the 2 metres that are left. It stops 30 centimetres past the hole, and the third putt fixes those 30 centimetres.

Each putt is small and imperfect. But each one corrects what the putts before it left. Together they reach the hole.

The problem: one fixed number for every product

Section titled “The problem: one fixed number for every product”

The global linear regression of the last lesson used the promotion flag. It learned one weight for it: “on promotion, add w units”. The same w for every product.

Here is a tiny constructed example. Two products, each seen on a normal day and on a promotion day:

Product On promotion Units sold Linear model Miss
slow no 4 0 4
slow yes 8 12 −4
fast no 20 24 −4
fast yes 40 36 4

A promotion doubles the sales of both products: +4 for the slow one, +20 for the fast one. The best linear model with the columns “fast” and “on promotion” is: units = 24 × fast + 12 × promotion. Its 12 units are too many for the slow product and too few for the fast one. No weights fix this, because the effect of a promotion depends on the product. Two features that act together like this have an interaction.

Larkfield’s promotions work like this. Their median uplift is ×1.87 (from Calendars, holidays and events): a multiplication, not an addition.

The idea: many small trees, each fixing the last errors

Section titled “The idea: many small trees, each fixing the last errors”

A decision tree predicts by asking yes-or-no questions: “Is the product a fast seller? Is it on promotion?” Each answer leads to a leaf with a number. A tree can ask about the promotion after it asked about the product, so it can give a different uplift to each kind of product.

One small tree is a weak forecaster. Gradient boosting builds many of them, like the golfer’s putts:

  1. Start with one number for every row: the mean.
  2. Compute the errors that are left.
  3. Fit a small tree that predicts those errors.
  4. Add a small step of that tree to the forecast.
  5. Repeat, hundreds of times.

The name says what it does. Boosting: weak models, one after the other, become a strong one. Gradient: each new tree follows the direction that reduces the error most. LightGBM is a library that does this fast on large tables. The machine learning course explains it in depth in Ensembles.

The same four rows. Real libraries take very small steps (here 0.05) and grow many trees. This example takes full steps, so that two trees are enough.

Product Promotion Sold Start (mean) Error left Tree 1: fast? After tree 1 Error left Tree 2: fast? then promotion? After tree 2
slow no 4 18 −14 −12 6 −2 −2 4
slow yes 8 18 −10 −12 6 2 +2 8
fast no 20 18 2 +12 30 −10 −10 20
fast yes 40 18 22 +12 30 10 +10 40

Follow the slow product on promotion, the second row. The start is the mean of the four rows: (4 + 8 + 20 + 40) ÷ 4 = 18. Tree 1 asks one question, “fast seller?”, and predicts the mean error of each group: −12 for the slow rows. So the forecast becomes 18 − 12 = 6. The error left is 8 − 6 = 2. Tree 2 asks two questions, “fast?” then “on promotion?”, and predicts +2 for this leaf. The forecast is 6 + 2 = 8.

Tree 2 gives +2 to the slow product on promotion and +10 to the fast one: the interaction that the linear model could not learn. On real data, the many small steps keep the model from learning the noise.

Larkfield: one LightGBM model for all products

Section titled “Larkfield: one LightGBM model for all products”

Priya trains LightGBM with mlforecast, with the inputs of the course table. They are the lags of 1, 7, 14 and 28 days and the means of the last 7 and 28 days. One more input is the mean of lags 7 to 10: the 4 days that end a week back. The calendar adds the weekday, the month and the day of the year. The known inputs are the price, the promotion flag and three holiday flags.

import lightgbm as lgb
from mlforecast import MLForecast
from mlforecast.lag_transforms import RollingMean
LGB = dict(n_estimators=400, learning_rate=0.05, num_leaves=63, min_child_samples=20, subsample=0.8,
subsample_freq=1, colsample_bytree=0.8, random_state=35, n_jobs=2, verbose=-1)
def make_model(lags, lag_transforms):
return MLForecast(models={"lgbm": lgb.LGBMRegressor(**LGB)}, freq="D", lags=lags, lag_transforms=lag_transforms,
date_features=["dayofweek", "month", "dayofyear"], num_threads=1)
RECURSIVE = dict(lags=[1, 7, 14, 28], lag_transforms={1: [RollingMean(7), RollingMean(28)], 7: [RollingMean(4)]})

n_estimators=400 is the number of trees. learning_rate=0.05 is the size of each step. num_leaves=63 limits the size of each tree. These are the settings of the course table, not tuned values.

At the 13 origins of the course backtest, the notebook prints:

method horizon MASE WAPE % bias % pinball fit s cost $
LightGBM, recursive 14 0.816 40.7 0.7 12.54 14749

LightGBM beats ETS (41.3%) and both linear regressions (41.7% and 42.6%) on WAPE, and its order costs 14,749 dollars against 16,578 for ETS. On the 154 promotion days, its bias is +5.2%. ETS was −43.5% there.

The explorer has the methods that you have met so far. Move the origin and compare LightGBM with ETS and the baselines on the parasol.

Explore

LightGBM on the parasol

Sold in the 14 days
101
Forecast for the 14 days
114
WAPE at this origin
44.8%
WAPE, all 13 origins
49.3%

At the origin 8 Jun 2025, LightGBM (recursive) forecasts 114 units for the next 14 days. 101 were sold. Its WAPE here is 44.8%: it is too high. Over all 13 origins, LightGBM (recursive) has a WAPE of 49.3%: number 1 of 5 methods. It beats the baseline, seasonal naive (66.1%).

13 Apr 202511 May 20258 Jun 20256 Jul 20253 Aug 202531 Aug 202528 Sep 202526 Oct 202523 Nov 202521 Dec 202518 Jan 202615 Feb 202615 Mar 2026
13 Apr 202511 May 20258 Jun 20256 Jul 20253 Aug 202531 Aug 202528 Sep 202526 Oct 202523 Nov 202521 Dec 202518 Jan 202615 Feb 202615 Mar 2026
  • Units sold
  • Forecast of the chosen method
  • Baseline: seasonal naive
0102030Units sold per day12 May19 May26 May2 Jun8 Jun15 Jun22 Junorigintest: 14 daysSold before the originSold 9 Jun 2025: 1Sold 10 Jun 2025: 5Sold 11 Jun 2025: 5Sold 12 Jun 2025: 4Sold 13 Jun 2025: 3Sold 14 Jun 2025: 9Sold 15 Jun 2025: 7Sold 16 Jun 2025: 4Sold 17 Jun 2025: 10Sold 18 Jun 2025: 5Sold 19 Jun 2025: 3Sold 20 Jun 2025: 12Sold 21 Jun 2025: 20Sold 22 Jun 2025: 13seasonal naive: 118LightGBM (recursive): 114
0102030Units a day18 May8 Jun22 Junorigintest: 14 daysSold before the originSold 9 Jun 2025: 1Sold 10 Jun 2025: 5Sold 11 Jun 2025: 5Sold 12 Jun 2025: 4Sold 13 Jun 2025: 3Sold 14 Jun 2025: 9Sold 15 Jun 2025: 7Sold 16 Jun 2025: 4Sold 17 Jun 2025: 10Sold 18 Jun 2025: 5Sold 19 Jun 2025: 3Sold 20 Jun 2025: 12Sold 21 Jun 2025: 20Sold 22 Jun 2025: 13seasonal naive: 118LightGBM (recursive): 114
Show every method, ranked
Every method, scored on all 13 origins (lower WAPE is better)
RankMethodFamilyWAPE at this originWAPE, all 13 origins
1LightGBM (recursive)machine learning44.8%49.3%
2AutoARIMAstatistical51.7%62.6%
3moving average (28 days)baseline101.3%64.9%
4ETSstatistical45.5%65.1%
5seasonal naivebaseline74.3%66.1%
The parasol P025 at the 13 origins of the course backtest, 14 days each: the real sales, the chosen method's forecast and the seasonal naive baseline (dashed). Every forecast is measured; the WAPE adds up all 182 forecast days.
Text description of the explorer

The explorer shows the parasol’s daily sales: the four weeks before the chosen origin (grey line) and the 14 days after it (grey dots). It adds the chosen method’s forecast (blue) and the seasonal naive forecast (dashed orange). The controls choose the origin (13 April 2025 to 15 March 2026) and one of five methods. At the origin of 8 June 2025, 101 parasols were sold; LightGBM forecast 114 (WAPE 44.8% at this origin) and ETS 92 (45.5%). Over all 13 origins, the WAPE of each method is: LightGBM (recursive) 49.3%, AutoARIMA 62.6%, moving average (28 days) 64.9%, ETS 65.1% and seasonal naive 66.1%.

On the parasol, LightGBM wins clearly. Across the 50 products, the lead is much smaller, and ETS still wins on 21. The last lesson of this module looks at that.

Kenji’s order covers 14 days. The model above uses lag1, but on day 5 of the order, “yesterday” has not happened yet. There are three honest answers.

Two panels. Recursive: the values known up to Sunday feed one model; the model forecasts day 1, day 2, day 3 up to day 14, and arrows show each forecast feeding the next day. Direct: the values known up to Sunday feed 14 separate models, model 1 for day 1 up to model 14 for day 14, with no arrows between the days.
A recursive forecast uses one model again and again, with its own forecasts as inputs. A direct forecast trains one model per day ahead, and each one uses only what is known on Sunday.
Text description of the diagram

The top panel, recursive, shows a box “known up to Sunday”, then one model that feeds day 1, day 2, day 3 and day 14. Arrows say that each forecast becomes the next day’s lag 1, so an error on day 1 can pile up. The bottom panel, direct, shows the same days, each with its own model, model 1 to model 14. Each model sees only what is known on Sunday.

  • Recursive (mlforecast’s default): one model. Its forecast for day 1 becomes lag1 for day 2, and so on. It is fast, but an error on day 1 is carried into day 2.
  • Lags of 14 days and more: one model with only inputs that are at least 14 days old. Nothing needs to be filled in, but the model never sees the last two weeks.
  • Direct: max_horizon=14 trains 14 models. The model for day 5 learns from rows where the inputs are 5 days older than the target. No forecast is fed back, but the fit takes much longer.

The course table has all three:

  • Statistical method
  • Machine learning
The course backtest table (50 products, 13 origins, 14 days). The direct forecast has the lowest order cost of the whole table, at about ten times the fit time of the recursive one. Fit times vary by about 30% between runs.
Text description of the chart

Three panels. WAPE: ETS 41.3%, LightGBM recursive 40.7%, lags of 14 days and more 40.9%, direct 40.7%. Cost of the 14-day order: ETS 16,578 dollars, recursive 14,749, lags of 14 days and more 14,629, direct 13,711. Fit time for the 13 origins: ETS about 46 seconds, recursive about 16, lags of 14 days and more about 12, direct about 156.

The three have almost the same WAPE, but their costs differ. The direct forecast costs 1,038 dollars less per order than the recursive one.

Be careful with small tests. To save time, the notebook runs the direct forecast at the last 4 origins only. There, it is the most expensive of the three:

method horizon MASE WAPE % bias % pinball fit s cost $
lags >= 14, last 4 cut-offs 14 0.652 38.9 -1.4 12.49 9174
recursive, last 4 cut-offs 14 0.682 41.9 0.2 12.54 9756
direct, last 4 cut-offs 14 0.696 42.0 2.7 93.53 9926

(The fit s of the first two rows is the time of all 13 origins.) Four origins are four winter and early-spring windows, and the order of the methods changes between origins. Compare them on all 13.

A rolling mean is LightGBM’s most useful feature, as the next section shows. It is also the easiest one to make wrong.

The parasol's sales from Monday 2 to Monday 9 June 2025: 13, 8, 9, 8, 8, 8, 5, 1. In the leaky row, a bar covers Tuesday 3 to Monday 9: the mean, 6.7, includes Monday's own sales. In the honest row, the bar covers Monday 2 to Sunday 8: the mean, 8.4, stops the day before.
A 7-day rolling mean for Monday 9 June. With the day itself inside the window, the feature contains the answer. A shifted window stops the day before.
Text description of the diagram

Two rows show the parasols sold from Monday 2 to Monday 9 June 2025: 13, 8, 9, 8, 8, 8, 5 and 1. Monday 9, in orange, is the day to forecast. In the leaky row, the window covers Tuesday 3 to Monday 9, so its mean, 6.7, includes Monday’s own sale of 1. In the honest row, the window covers Monday 2 to Sunday 8, and its mean is 8.4.

s.rolling(7).mean() in pandas includes the current row. Tomás computes it on the whole series before the backtest and gives it to LightGBM, with the weekday and the promotion flag. In the backtest:

Method MASE WAPE % Cost $ per order
LightGBM with the leaky rolling mean 0.671 33.5 4,899
LightGBM, recursive (honest) 0.816 40.7 14,749

The leaky model has a third of the honest model’s cost, and that is the warning sign. The model has learned to copy an input that will not exist in the order week.

mlforecast avoids this trap: lag_transforms={1: [RollingMean(7)]} computes the mean from lag 1, so it stops the day before. When you write a feature in pandas, shift first, then roll.

LightGBM can say how much each feature helped. The gain of a feature is how much the splits on it reduced the error, added up over all trees. Here is the share of each feature in the model trained at the last origin:

The share of each feature in the total gain, LightGBM recursive at the last origin (15 March 2026). The two recent means carry three quarters of the gain. The promotion flag is not in the top 10.
Text description of the chart

The top 10 features by share of the gain: mean of the last 7 days 52.4%, mean of the last 28 days 22.5%, yesterday (lag 1) 6.7%, day of the year 4.3%, mean of the same weekday, 4 weeks 4.1%, lag 28 1.8%, lag 7 1.8%, lag 14 1.8%, weekday 1.5%, price 1.2%.

The model mostly asks “how much has this product sold lately?”, then adjusts. Tomás sees no promotion flag in the top 10 and concludes: “Promotions don’t matter.” That is wrong. Promotion days are rare, so splits on them reduce the total error less, even though they matter a lot on those days. Importance tells you what the model used, not what causes sales. The machine learning course explains this limit in Explain predictions.

Use a global gradient boosting model when you have many related series and known future inputs. It is strongest when effects interact, such as a promotion that multiplies sales.

Its costs:

  • Settings. The number of trees, the step size and the tree size change the result. Tuning them on the backtest makes the score look better than it will be, as Test like the future explained (“many looks”).
  • Fit time. Seconds for the recursive model, minutes for the direct one, at every refit.
  • No range. LightGBM gives one number per day. Module 7 adds a range.
  • No values beyond what it has seen. A tree cannot forecast higher than its highest leaf. If a product grows to sales it never had before, ETS can follow the trend; a tree cannot.
  • Trust. Kenji can read an ETS forecast as “recent level plus the weekly pattern”. A model of 400 trees is harder to explain.

Common mistake

A rolling mean that includes the day

Symptom: a new LightGBM model has a WAPE of 33.5% and an order cost of 4,899 dollars. The honest model has 40.7% and 14,749.

Cause: its rolling mean was computed with rolling(7) on the whole series before the backtest. Each day’s feature contains that day’s sales, which are unknown at forecast time.

Fix: build rolling features from lags (shift first, then rolling), or with mlforecast’s lag_transforms. Recompute the features inside the backtest, from each origin’s training data only.

Consequence in AI work: a leaky model wins every backtest and then fails in production. When a model beats every other method by a wide margin, look for leakage before you celebrate.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the helper cell. The data cell ends with the Ready: line.
  3. In section 1, predict which feature LightGBM uses most. Then run the cell. You see rolling_mean_lag1_window_size7 first, with 52.4.
  4. In section 2, predict if LightGBM beats ETS. Then run the backtest. You see WAPE % 40.7 and cost 14749.
  5. In section 3, run the three set-ups (about a minute). You see the lags of 14 days and more with WAPE % 40.9, then three rows for the last 4 origins.
  6. In section 4, predict if the leaky model looks better or worse. Then run it. You see WAPE % 33.5.
  7. In section 5, write your answer in the Your turn cell, then run the check cell.

ETS missed 43.5% of the units on the 154 promotion days. Before you run anything, predict: is LightGBM too high or too low on those days? In the notebook, section 5, keep the promotion days of fc_rec. Compute their bias in % and put it in promo_bias_lgbm.

Show the answer
p = fc_rec.merge(sales[["unique_id", "ds", "on_promotion"]], on=["unique_id", "ds"])
p = p[p["on_promotion"] == 1]
promo_bias_lgbm = round(float(100 * (p["yhat"].sum() - p["y"].sum()) / p["y"].sum()), 1)
print(promo_bias_lgbm, "% on", len(p), "promotion days")
5.2 % on 154 promotion days

LightGBM is 5.2% too high on the promotion days, close to the global linear regression (+5.9%) and far from ETS (−43.5%). It sees the promotion flag. On the other days, it is only 0.5% too high.

Check your understanding

Explain with the tiny example why a linear model cannot learn a promotion that doubles sales, and how two boosted trees can.

Show the answer

The slow product sells 4, or 8 on promotion; the fast one sells 20, or 40. The promotion adds 4 units to one and 20 to the other. A linear model has one weight for “on promotion”, so it adds the same number to both. It adds 12 units: too many for the slow product and too few for the fast one. Gradient boosting starts from the mean (18) and adds trees that predict the errors left. Tree 1 asks “fast seller?” (−12 or +12). Tree 2 asks “fast?” then “on promotion?” and gives +2 to the slow product on promotion and +10 to the fast one. A tree can give each kind of product its own uplift.

Check your understanding

Name the three ways to forecast 14 days ahead with LightGBM, and give one advantage and one cost of the direct forecast, with Larkfield's numbers.

Show the answer

Recursive: one model, which uses its own forecasts as lags for the next days. Lags of 14 days and more: one model with inputs that are all known for the whole order. Direct: one model per day ahead (14 models), each using only what is known at the origin. The direct forecast had the lowest order cost of the course table, 13,711 dollars against 14,749 for the recursive one. It cost about ten times the fit time (about 156 seconds against 16 for the 13 origins). And at the last 4 origins alone, it was the most expensive of the three, so a small test can mislead.

Check your understanding

Predict: a colleague's new feature brings LightGBM's WAPE from 40.7% to 33.5%. What do you check first, and how?

Show the answer

Check for leakage first: a jump that large from one feature is the sign of a feature that contains the target. Look at how the feature is computed. A rolling mean with rolling(7) on the whole series, without a shift, includes the day itself. Recompute it from lags (shift, then roll) inside the backtest, from the training data of each origin. On Larkfield, the leaky rolling mean gave 33.5% and an order cost of 4,899 dollars. Without the leak, the model is back at 40.7% and 14,749.

Recap

  • Gradient boosting adds many small decision trees; each one predicts the errors that the trees before it left. Trees learn interactions, such as a promotion that doubles sales.
  • One global LightGBM model for the 50 products: WAPE 40.7%, order cost 14,749 dollars, +5.2% on promotion days.
  • Many days ahead: recursive (fast), lags of 14 days and more, or direct: one model per day. Direct has the lowest cost of the table, 13,711 dollars, at about ten times the fit time.
  • A rolling mean that includes the day is leakage: WAPE 33.5% and a cost of 4,899 dollars that can never happen. Shift first, then roll.
  • Feature importance says what the model used (the recent means: three quarters of the gain), not what causes sales.

You can now train and backtest a global machine learning forecaster. But Tomás has a question: why train at all, when a pretrained model can forecast any series? Next, you run a foundation model with no training and measure what it costs.

Next: Deep and foundation models