Skip to content
Français

When not to build a model

  • Lesson 17
  • 30 min
  • Module 5: Machine learning forecasters

Where we are

In the last three lessons, the backtest table got machine learning and foundation model rows. The most accurate method was not the cheapest, and the cheapest took ten times longer to fit. In this lesson, we read the whole table for cost, speed and trust, and see where an LLM fits: the forecast note, not the numbers. Think first of how a family chooses a car.

What you will be able to do

Choose a forecasting method the way Kenji pays for it. Look at the cost of the decision, the work to run the method, and how far people can trust it. Sometimes the simple method wins. And an LLM can write the weekly note about the forecast, but it must never make the numbers.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It sorts the backtest table three ways and compares ETS, the seasonal naive forecast and LightGBM product by product. Then it checks every number in ten forecast notes that an LLM drafted. It makes no model calls: the notes were recorded once. Its setup cells install the packages and download the data, so it needs no local setup. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

Nothing: the notebook runs in Colab or Kaggle. It needs no API key.

Tested in Jupyter on macOS, with Python 3.12, pandas 2.3.3, statsforecast 2.1.1, mlforecast 1.1.0 and LightGBM 4.7.0.

A family needs a car for the school run and the weekly shopping. The magazine’s test says that one sports car is the fastest on the track. The family does not buy it. It costs more to buy, to insure and to repair, and it has no room for the shopping. A small car is slower on the track and better for their week.

The test on the track is real. It answers a different question. A good choice reads every column of the test: speed, price, running costs, and what the car is for.

The problem: the winner changes with the column

Section titled “The problem: the winner changes with the column”

The backtest table of the orders now has 13 rows. Three people read it, and each one finds a different winner.

  • Tomás sorts by WAPE: Chronos-Bolt wins with 38.7%.
  • Priya sorts by cost: the direct LightGBM wins with 13,711 dollars per order.
  • Mei sorts by fit time: the naive forecast wins with 0.02 seconds for all 13 origins.

Each of them is right about their column. None of them has answered Kenji’s question: “Which method should run every Monday?”

The chart puts every row in the order of Kenji’s cost, with the WAPE next to it.

  • Baseline
  • Statistical method
  • Machine learning
  • Foundation model
All 13 methods of the course backtest table, cheapest order first. Grey: baselines; blue: statistical methods; orange: machine learning; aqua: the foundation model. Simple and complex methods are mixed: the seasonal naive forecast costs less than Theta and the global linear regression. Costs are the course's assumptions.
Text description of the chart

The 13 methods, cheapest order first, with the cost per order and the WAPE: LightGBM direct 13,711 dollars (40.7%); LightGBM lags ≥ 14 14,629 dollars (40.9%); LightGBM recursive 14,749 dollars (40.7%); Chronos-Bolt 14,750 dollars (38.7%); linear regr., per product 15,137 dollars (41.7%); AutoARIMA 16,257 dollars (41.4%); ETS 16,578 dollars (41.3%); seasonal naive 16,739 dollars (51.3%); Theta 16,812 dollars (41.5%); linear regr., global 17,264 dollars (42.6%); moving average 28 d 19,444 dollars (43.5%); naive 28,261 dollars (53.2%); drift 28,570 dollars (53.8%).

Every value of the table, in the same order:

Method Kind MASE WAPE % bias % fit s (13 origins) cost $ per order
LightGBM direct machine learning 0.813 40.7 −0.6 156 13,711
LightGBM lags ≥ 14 machine learning 0.828 40.9 −0.7 12 14,629
LightGBM recursive machine learning 0.816 40.7 +0.7 16 14,749
Chronos-Bolt foundation model 0.784 38.7 −5.6 2 14,750
linear regr., per product machine learning 0.835 41.7 −1.4 13 15,137
AutoARIMA statistical method 0.834 41.4 +1.4 996 16,257
ETS statistical method 0.832 41.3 +0.1 46 16,578
seasonal naive baseline 1.034 51.3 −0.5 0.05 16,739
Theta statistical method 0.837 41.5 +0.1 42 16,812
linear regr., global machine learning 0.866 42.6 −0.1 0.33 17,264
moving average 28 d baseline 0.875 43.5 −0.1 0.11 19,444
naive baseline 1.065 53.2 +7.4 0.02 28,261
drift baseline 1.075 53.8 +7.8 0.04 28,570

Read the middle of the table. The seasonal naive forecast, “the same weekday last week”, costs 16,739 dollars. That is less than Theta and the global linear regression, and only 161 dollars more than ETS. AutoARIMA costs about the same as ETS and needed about 1,000 seconds instead of 46. And the 28-day moving average has a better WAPE than the seasonal naive forecast but costs 2,705 dollars more per order.

The idea: a complex model must earn its cost

Section titled “The idea: a complex model must earn its cost”

A complex model is worth it only when its gain is larger than everything it costs. The table shows one part of the cost: the fit time. The rest is not in the table:

  • Running it: a weekly job, its packages, its memory, and someone who fixes it when it breaks. Chronos-Bolt even needed its own process.
  • Trust: Kenji must understand a forecast well enough to overrule it when he knows something the model does not.
  • Change: a feature set or a pretrained model has more ways to break when the data changes.

So the question is never “which method is best?”. It is “is the gain over the simple method worth what the complex method costs here?”

A worked example: is LightGBM worth it for Larkfield’s orders?

Section titled “A worked example: is LightGBM worth it for Larkfield’s orders?”

Take the cheapest simple method and the cheapest complex one:

Seasonal naive LightGBM, direct
Cost per order 16,739 dollars 13,711 dollars
Fit time, 13 origins 0.05 seconds about 156 seconds
Code to run one line features, holiday flags, 14 models

The gain is 3,028 dollars per order for the 50 products. A few minutes of fit time a week is cheap. Here, the complex model is worth building: the backtest says so in dollars, under the course’s cost assumptions. Doing this sum is the point of the lesson. The next cases show where the same sum goes the other way.

The table adds up 50 products. The notebook compares the WAPE of each product:

The 50 products of the course backtest, WAPE over the 13 origins. Top: the method with the lowest WAPE among seasonal naive, ETS, AutoARIMA, LightGBM (recursive) and Chronos-Bolt. Bottom: how often a simple method beats LightGBM on its own.
Text description of the chart

Lowest WAPE per product: Chronos-Bolt on 28 products, LightGBM (recursive) on 16, ETS on 3, AutoARIMA on 3. ETS beats LightGBM on 21 of the 50 products; the seasonal naive forecast beats LightGBM on 3.

ETS beats LightGBM on 21 of the 50 products. The seasonal naive forecast beats it on 3: P013, P029 and P075. LightGBM wins in total because it wins big on products such as the parasol (49.3% against 65.1% for ETS). For the other products, its lead is small or negative.

Module 6 forecasts 140 slow product-region series: products that sell nothing on 70% of the days or more. Intermittent demand teaches them; here you need only the result. There, a forecast of always 0 has the best MASE of all (0.599). But it orders nothing, so its order costs 4,371 dollars. The cheapest order comes from the plain seasonal naive forecast (2,885 dollars). Croston’s method, a method built for slow sellers, costs 3,733.

For Grace’s roster, Theta has the best ticket WAPE (27.1%). From forecast to order and roster sets the roster from a high quantile of the forecast. Then MSTL gives the cheapest roster: 3,718 dollars a week, against 5,845 for Theta, whose range is far too wide. MSTL is a statistical method that fits in about 3 seconds.

Signs that a complex model is not worth it

Section titled “Signs that a complex model is not worth it”
  • Its gain over the seasonal naive forecast or ETS, in the decision’s cost, is small next to its running costs.
  • It wins on the error measure but not on the cost (Chronos-Bolt here, or Theta for the roster).
  • It wins in total but loses on many products.
  • You have few series or short history, so there is little to learn.
  • Nobody on the team can explain its forecast to the person who acts on it.
  • It needs its own process, a GPU or a large download that the team cannot run.

Every Monday, Omar wants a short note: what the plan orders, how many agent hours it staffs, and what to watch. Priya writes it by hand. A large language model (LLM) can draft it in seconds.

Tomás goes one step further: “Why not ask the LLM for the forecast itself?” Because it fails every test of this course. An LLM writes text that is likely, not numbers fitted to Larkfield’s 400,000 rows. Nobody can backtest it in the same way, it can give a different answer to the same question, and it cannot show its cost column. The forecasting job computes the numbers. The LLM only turns them into sentences.

Four boxes from top to bottom: the forecasting job computes the order, the roster and the alarms; the week's numbers as JSON; the LLM chat-small writes 80 to 120 words and is asked to use only these numbers; the number check, code with no LLM, requires every number in the note to be in the JSON. At the bottom, two outcomes: passes, a person reads it, though the words can still be wrong; fails, fix the numbers before anyone reads it.
The numbers come from the backtested job. The LLM writes the sentences. Code checks every number in the note before a person reads it.
Text description of the diagram

Four boxes from top to bottom. The forecasting job computes the order, the roster and the alarms, with backtested methods and no LLM. The week’s numbers go out as JSON, for example 13,468 units to order, 382 agent hours and an alarm of −23% and −56%. The LLM, chat-small, writes 80 to 120 words and is asked to use only these numbers. A number check, written in code with no LLM, requires every number in the note to be one of the numbers in the JSON. Then there are two outcomes. If the note passes, a person reads it, because the words can still be wrong. If it fails, someone fixes the numbers first.

The course recorded 10 notes once, from Azure’s chat-small deployment, for 10 weeks of the forecasting job of Module 8. They used 3,831 input and 4,546 output tokens, about US$0.003 in total at the course’s price table (prices change: check the current page). The check finds every number in the note and looks for it in the numbers that the LLM received:

NUMBER = re.compile(r"(?<![\w.])[-+]?\d[\d,]*(?:\.\d+)?%?") # 12,839 or -23% or 0.5
def check(note, facts):
"""The numbers in the note that are not in the facts (within 1%). 1, 2, 7 and 14 are always allowed."""
allowed = {value(m) for m in NUMBER.findall(json.dumps(facts))} | {1.0, 2.0, 7.0, 14.0}
return [m for m in NUMBER.findall(note) if not any(abs(value(m) - a) <= 0.01 * max(abs(a), 1) for a in allowed)]

The output for the ten notes:

2025-03-10 passes
2025-03-17 NOT SUPPORTED: ['23%', '56%']
2025-03-24 passes
2025-03-31 passes
2025-04-07 passes
2025-04-14 passes
2025-04-21 passes
2025-04-28 passes
2025-05-05 passes
2025-05-12 passes

Nine notes pass. The note of 17 March 2025 does not. The notebook prints the first 115 characters of the alarm and the sentence of the note with the same numbers:

ALARM: BIAS ALARM plants-and-seeds: the forecast was too low 2 weeks in a row (-23% and -56%, weeks of 2025-03-03, 2025-03
NOTE: A bias alarm says the plants-and-seeds forecast was too low for 2 weeks in a row: by 23% and 56% in the weeks of 2025-03-03 and 2025-03-10

The LLM dropped the minus signs. In this sentence, “too low by 23%” still means the same. But the check cannot read meaning, and in a table or a shorter sentence, “+23%” and “−23%” are opposite advice: order more, or order less.

The check has a limit in the other direction too. The note of 10 March passes. Its first sentence says: “Larkfield’s plan for the week of 2025-03-10 is to handle 12,839 orders”. The plan says order_units: 12839: 12,839 units to order, not customer orders. The number is right; the word is wrong. A check of numbers does not check meaning. A person still reads every note before it goes out.

  • Money: a fraction of a cent per note.
  • The check: a few lines of code, run on every note.
  • A reader: someone who reads the note against the plan, because some mistakes pass the check.
  • Small surprises: chat-small is a reasoning model: it works through hidden steps before it writes its answer. The first recording call failed with an error, because it does not accept the usual max_tokens setting.

Common mistake

A dropped minus sign

Symptom: the note of 17 March says that the plants-and-seeds forecast was too low “by 23% and 56%”. The plan’s alarm says −23% and −56%. The number check flags the note: NOT SUPPORTED: 23%, 56%.

Cause: the LLM rewrote the numbers into its own sentence and dropped their signs. It was told to use only the numbers it got. An instruction is not a guarantee.

Fix: never send a note that fails the check. Ask for the numbers exactly as written in the plan. Or let the job write the numbers into fixed sentences, and the LLM only the rest. Then a person reads the note.

Consequence in AI work: a reader trusts a well-written note. A wrong sign in fluent text reverses the advice, and nobody notices without a check.

Guided practice: read the table, check the notes

Section titled “Guided practice: read the table, check the notes”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the helper cell. The data cell ends with the Ready: line.
  3. In section 1, predict if the lowest WAPE has the lowest cost. Then run the cell. You see three different winners.
  4. In section 2, predict on how many products ETS beats LightGBM. Then run the cell (about a minute). You see 21 of 50.
  5. In section 3, write your answer in the Your turn cell, then run the check cell.
  6. In section 4, predict if all 10 notes pass. Then run the check. You see one note flagged.
  7. Run the next two cells. Compare the alarm with the note, then read the note of 10 March.

Before you run anything, predict: on how many of the 50 products does “the same weekday last week” beat LightGBM? In the notebook, section 3, count them from per_product and put the count in naive_wins.

Show the answer
naive_wins = int((per_product["seasonal naive"] < per_product["LightGBM"]).sum())
print(naive_wins, per_product.index[per_product["seasonal naive"] < per_product["LightGBM"]].tolist())
3 ['P013', 'P029', 'P075']

On 3 products, a one-line forecast beats a 400-tree model. P029 is the slowest of the 50 products and P075 the best seller. So the size of a product does not predict where the simple method wins. Only the backtest tells you.

Check your understanding

Using the backtest table, explain why Kenji should not simply take the method with the lowest WAPE.

Show the answer

Kenji pays for the order, not for the daily error. Chronos-Bolt has the lowest WAPE (38.7%), but it is biased low (−5.6%) and its order costs 14,750 dollars. The direct LightGBM has a WAPE of 40.7% and the lowest cost, 13,711 dollars. The 28-day moving average has a better WAPE than the seasonal naive forecast (43.5% against 51.3%). But it costs 19,444 dollars against 16,739. The method must be chosen on the cost of the decision, then on fit time, running costs and trust.

Check your understanding

Give two cases from the course where a simple method beats a complex one, and one case where the complex method is worth its cost.

Show the answer

Simple wins: ETS beats LightGBM on 21 of the 50 products, and the seasonal naive forecast on 3. On the slow sellers, the seasonal naive forecast has the cheapest order (2,885 dollars), cheaper than Croston’s method (3,733). For the roster at a high quantile, MSTL is the cheapest (3,718 dollars a week) although Theta has the better WAPE. Complex is worth it for the 50 products’ orders. The direct LightGBM costs 3,028 dollars less per order than the seasonal naive forecast, for a few minutes of fit time.

Check your understanding

Why does Larkfield use an LLM for the weekly note but not for the forecast, and what can the number check not catch?

Show the answer

The forecast must be backtested, repeatable and scored on the cost of the decision. An LLM writes likely text, not numbers fitted to the data. So the backtested job computes the numbers. The LLM only turns them into sentences. Code checks that every number in the note is in the plan. 9 of 10 recorded notes passed; the note that dropped the minus signs of −23% and −56% was flagged. The check cannot catch a wrong meaning: the note of 10 March calls 12,839 units “orders” and passes. A person must still read the note.

Recap

  • The winner changes with the column. Chronos-Bolt wins by WAPE (38.7%). The direct LightGBM wins by cost (13,711 dollars). The naive forecast wins by fit time.
  • A complex model must earn its cost in the decision’s dollars, after running costs and trust. For the orders, LightGBM does: 3,028 dollars less per order than the seasonal naive forecast.
  • Simple wins elsewhere: ETS beats LightGBM on 21 of 50 products. On slow sellers, the seasonal naive forecast gives the cheapest order. MSTL gives the cheapest roster.
  • An LLM writes the note, never the numbers. Check every number in code: 9 of 10 notes passed, and one dropped its minus signs. Then let a person read the note.

You can now choose a forecaster by what it costs, not only by its error. But every forecast so far was for one product at a time. Larkfield’s products add up to categories, regions and a total, and the forecasts must agree. Next, you make forecasts at every level of the hierarchy add up.

Next: Hierarchies that must add up