People and forecasts
- Lesson 27
- 40 min
- Module 8: Run a forecast in production
Where we are
Our job watches its own error. People also change the forecast: Kenji raises an order before a promotion. In this lesson, we record each override with its reason, measure whether it helped, and check a drafted forecast note against the numbers. Think first of a taxi driver who does not follow the map.
Lesson 27 of 27 · Forecasting and Time Series: Predict What Comes NextPeople and forecasts
1. Forecasts serve decisions
2. Read a time series
3. Baselines and honest testing
4. Statistical forecasters
5. Machine learning forecasters
6. Hard series and many series
7. Uncertainty and the decision
8. Run a forecast in production
What you will be able to do
Let planners change the job’s numbers, but make every change visible: who changed what, and why. Then measure, with the real sales, whether each change saved money. Use a language model only for the words of the weekly note, and check its numbers against the plan.
Before you startWhat to know and to have ready
What you need to know
- Your job folder
forecast-joband its storeruns/with the spring replay, from Watch the forecast. - The cost of a stock order, from Too many parasols, too few staff. A lost sale costs 35% of the price, and a left-over item 25%. Both are assumptions of the course.
- A language model (LLM) writes text from a prompt. Building Reliable Applications with LLM APIs shows how to call one. This lesson makes no model call.
What you need on your computer
The same as in the last lesson. The replay in this lesson takes about 45 seconds. No account, no key and no money: the drafted notes were recorded once by the course team.
A taxi driver who does not follow the map
Section titled “A taxi driver who does not follow the map”A taxi company gives every driver a map app. The app chooses the route. Sometimes a driver does not follow it. She knows that the bridge is closed today, or that a school lets out at three. Sometimes she is right and saves ten minutes. Sometimes she is wrong.
The company has two bad choices. It can forbid drivers to leave the route, and lose what they know. Or it can let them do what they like, and never know who is right. A good company does a third thing. It lets drivers leave the route, asks them to say why, and later compares the trip time with the time the app predicted.
Kenji is that driver, and the forecasting job is the map.
The problem: changes that nobody can judge
Section titled “The problem: changes that nobody can judge”Kenji knows things that the data does not show. A warm week is coming. A supplier sells only in boxes of 80. So he often changes the job’s order before he sends it.
Until now, he changed the numbers in his own spreadsheet. The job’s plan said 14 parasols for the south region; the supplier got 21. Nobody wrote down why. Two weeks later, Tomás says: “Kenji’s changes make the forecast worse.” Kenji says: “My changes save us money.” Neither can show it. The job kept its own number, but not Kenji’s, and no reason.
You saw the same gap in the last lesson. The bias alarm told Priya that the job is too low on garden products every spring. Kenji knew that already. If his knowledge helps, the shop should keep it until the method can learn the season. If it does not help, the shop should know that too.
The idea: record the override, then measure it
Section titled “The idea: record the override, then measure it”An override is a person’s change to a number that the job produced. The job records each one as a row in an overrides file:
| Column | Example |
|---|---|
week |
2025-03-31 |
product_id, region |
P025, south |
order |
21 (the job said 14) |
reason |
warm weather: garden season starts early |
who |
Kenji |
The job places Kenji’s number, but it keeps its own number next to it. A reason is required: a row without one stops the job.
Fourteen days later, the sales are known. Then you can ask the question fairly: which order would have cost less? The answer is called forecast value added, or FVA:
FVA = cost of the job’s order − cost of the planner’s order
Both costs use what was really sold in the 14 days, with the course’s costs. A positive FVA means the override saved money. A negative FVA means it cost money.
Here is the override of the table, worked out in full:
Text description of the diagram
The diagram starts with the job’s order: P025 Parasol, south region, week of 31 March 2025, 14 units. An arrow goes to Kenji’s override: 21 units, with the reason “warm weather: garden season starts early”. A box shows what happened in the 14 days: 29 parasols sold. Two boxes compute the costs. The job’s order was 15 units short: 15 × 28.00 = 419.95 dollars. Kenji’s order was 8 units short: 8 × 28.00 = 223.97 dollars. The last box: forecast value added = 419.95 − 223.97 = +195.98 dollars. The override helped.
(The exact cost of a lost sale is 35% of 79.99 dollars, 27.9965 dollars. So 15 lost sales cost 419.95 dollars.)
A left-over item counts too. In the week of 12 May, Kenji raised the coast order for the same parasol from 62 to 87: “weather forecast: a hot weekend”. Only 38 were sold. The job’s order left 24 over: 24 × 20.00 = 479.94 dollars. Kenji’s left 49 over: 979.88 dollars. FVA: −499.94 dollars. That override cost money.
Add the planner’s files
Section titled “Add the planner’s files”-
In
forecast-job, with the virtual environment active, copy the planner’s folder and the last two test files from the snapshotend-of-m08. Your project then matches the snapshot.Terminal window S=../nextia-learning-labs/C35/forecast-job/end-of-m08cp -R "$S/planner" .cp "$S/tests/test_overrides.py" "$S/tests/test_note.py" tests/Terminal window $S = "..\nextia-learning-labs\C35\forecast-job\end-of-m08"Copy-Item -Recurse "$S\planner" .Copy-Item "$S\tests\test_overrides.py", "$S\tests\test_note.py" tests\ -
Run the tests:
Terminal window python -m pytest -q25 passed in 6.08s -
Open
planner/overrides_replay.csvin VS Code. It has Kenji’s 10 overrides in the spring quarter. They are a situation we planted: the course team wrote them to show typical reasons. The header and the first two rows:week,product_id,region,order,reason,who2025-03-17,P186,north,37,warm weather: heaters will sell less,Kenji2025-03-24,P002,north,9,warm weather: garden season starts early,Kenji -
Replay the spring again.
replayreads this file by itself when it exists.Terminal window python -m forecast_job replay --start 2025-03-03 --weeks 12Week 2025-03-03 is already planned with the same inputs (ec8bedf4e7811156). Nothing to do.Week 2025-03-10 is already planned with the same inputs (244b2032b74c7443). Nothing to do.Check 2025-03-03: all categories +5.7%, largest plants-and-seeds -22.8%, 0 alarmsWeek 2025-03-17 was planned with other inputs (4043b1971ed95e4d, now d515abaae5bd43de). Run again with --force to replace the plan (the old one is kept).The job stopped at 17 March. The overrides are part of the fingerprint, and that week now has one. This is the store of The forecasting job at work: it never replaces a plan without being asked.
-
The overrides are what Kenji really sent, so replace the plans:
Terminal window python -m forecast_job replay --start 2025-03-03 --weeks 12 --forceThe lines of the weeks with overrides now end with
1 overridesor2 overrides, for example:Week 2025-03-31: 16510 units to order (expected cost 35006 dollars), 453 agent hours (expected cost 3836 dollars), 0 warnings, 1 overrides. Plan: runs/2025-03-31…Replayed 12 weeks from 2025-03-03: 17 bias alarms, 0 weeks stopped.Weeks without an override print
Nothing to do., as before. The alarms do not change: they check the forecast, and an override changes only the order. Each replaced plan is kept inruns/<Monday>/replaced-<fingerprint>/.
Did Kenji’s overrides help?
Section titled “Did Kenji’s overrides help?”The command fva finds every override in the store whose 14 days are over, and computes its FVA:
python -m forecast_job fva week product_id region reason who job_order override sold job_cost override_cost fva_usd2025-03-17 P186 north warm weather: heaters will sell less Kenji 73 37.0 23 237.37 66.46 170.912025-03-24 P001 north warm weather: garden season starts early Kenji 7 13.0 15 131.57 32.89 98.682025-03-24 P002 north warm weather: garden season starts early Kenji 5 9.0 11 56.68 18.89 37.792025-03-31 P025 south warm weather: garden season starts early Kenji 14 21.0 29 419.95 223.97 195.982025-04-07 P055 north Easter weekend Kenji 1 2.0 3 2.09 1.05 1.042025-04-14 P117 north gut feeling Kenji 1 2.0 1 0.00 8.25 -8.252025-04-14 P141 south supplier minimum order of 80 Kenji 57 80.0 58 26.25 412.44 -386.192025-04-28 P039 north warm weather: garden season starts early Kenji 12 21.0 44 739.09 531.22 207.872025-05-05 P118 coast gut feeling Kenji 1 2.0 0 11.00 22.00 -11.002025-05-12 P025 coast weather forecast: a hot weekend Kenji 62 87.0 38 479.94 979.88 -499.9410 overrides: 6 helped, 4 did not. Forecast value added: -193 dollars in total. count sumreasonEaster weekend 1 1.0gut feeling 2 -19.0supplier minimum order of 80 1 -386.0warm weather: garden season starts early 4 540.0warm weather: heaters will sell less 1 171.0weather forecast: a hot weekend 1 -500.0In total, the 10 overrides cost 193 dollars more than the job’s orders would have. So Tomás is right? Not quite. The total hides the story. Look at it by reason:
Text description of the chart
Forecast value added by reason, from the best to the worst: warm weather: garden season starts early (4) +540 dollars; warm weather: heaters will sell less (1) +171 dollars; Easter weekend (1) +1 dollars; gut feeling (2) -19 dollars; supplier minimum order of 80 (1) -386 dollars; weather forecast: a hot weekend (1) -500 dollars. The total of all 10 overrides is −193 dollars.
Each reason tells something different:
- Warm weather, the season (+540 dollars in 4 overrides, and +171 for the heaters). Kenji was right every time. He knew the yearly season that the job’s method cannot see: the cause of the bias alarms in the last lesson. Keep these overrides until the job has a method with a yearly season.
- A hot weekend (−500 dollars). A weather forecast for one weekend is a one-time event. Kenji raised a 14-day order for it, and most of the extra parasols stayed in the store.
- Supplier minimum order of 80 (−386 dollars). This is not a view about demand. The supplier sells P141 duvets only in boxes of 80. The FVA here is the cost of that rule: useful when Kenji talks to the supplier. Record it, but do not count it against his judgment.
- Gut feeling (−19 dollars in 2) and Easter weekend (+1 dollar). Small orders, small effects. “Gut feeling” is a reason that teaches nothing; ask for a better one.
So the useful answer for Omar is not “−193 dollars”. It is: “Kenji’s season overrides saved 711 dollars. One weekend override cost 500. The supplier’s rule cost 386.” Ten overrides are few, so treat this as a first look, not a proof. Measure again after the next quarter.
The note for the manager
Section titled “The note for the manager”Every Monday, Omar, the product manager, wants a short note: what we order, how many agent hours, and what to watch. Writing it takes Priya twenty minutes. A language model can draft it in seconds. But a model can also write a number that is not in the plan, and a wrong number in a note becomes a wrong decision.
The rule is simple: the model writes the words; the plan gives the numbers; a check compares them; a person reads the result. You saw this check on the ten recorded notes in When not to build a model. Here, the job runs it on its own plans.
Text description of the diagram
The diagram starts with the plan in runs/Monday/: the order, the roster, the warnings and the alarms. An arrow goes to the facts, which the command note –week prints: the numbers that a note may quote. The facts go to a drafted note: a small LLM writes the sentences, recorded once. Then a decision, the number check: is every number of the note in the facts? A dotted arrow shows that the check also reads the facts. If no, the result is NOT SUPPORTED: fix these numbers before you send. If yes, a person reads the note, because the check cannot see wrong words. Then the note is sent to the manager.
The course team sent the facts of 10 weeks to a small hosted LLM once: the deployment chat-small on Azure. Its 10 drafts are recorded in planner/notes_recorded.json. The 10 calls used 3,831 tokens in and 4,546 out, for about US$0.003 in total. This project makes no model calls: note reads the recorded drafts.
The check is in forecast_job/note.py. check_note() finds every number in the text with a regular expression (a search pattern for text), such as 15,344, 2025-03-24 or 7%. Each number must match a fact within 1%. Small counting words, 1, 2, 7 and 14, are always allowed.
A note that passes
Section titled “A note that passes”python -m forecast_job note --week 2025-03-24The command prints the facts as JSON, then the draft, then the check. The facts start like this:
{ "week": "2025-03-24", "order_units": 15344, "order_value_usd": 610177, "order_expected_cost_usd": 33596, "agent_hours": 382, …And the end of the output:
For the week of 2025-03-24, the plan orders 15,344 units, with an expected order value of $610,177. It staffs 382 agent hours. Plants and seeds, bedding and textiles, and kitchen are the leading categories. The peak is forecast for 2025-03-25 at 11:00, with 7 agents. There are 2 overrides and no data warnings. Bias alarms flag heating and decor as too high for 2 weeks in a row, and plants and seeds as too low for 2 weeks in a row; investigate possible causes before the next order. The forecast cannot know whether actual demand will match the plan or what may cause differences.
Check: every number is in the plan.Every number is in the facts. Even so, read it: “expected order value” is not right. 610,177 dollars is the value of the order. The expected cost is 33,596 dollars, and the note does not give it.
A note that fails
Section titled “A note that fails”python -m forecast_job note --week 2025-03-17… A bias alarm says the plants-and-seeds forecast was too low for 2 weeks in a row: by 23% and 56% in the weeks of 2025-03-03 and 2025-03-10. …
Check: NOT SUPPORTED by the plan: 23%, 56%. Fix these before you send the note.This is the note with the dropped minus signs from When not to build a model. The check cannot know that “too low by 23%” still means the right thing, so it stops. That is on purpose: a check that guesses would also let a real mistake pass. Priya writes −23% and −56%, or reads the sentence and decides that it is right.
A note that passes and is still wrong
Section titled “A note that passes and is still wrong”python -m forecast_job note --week 2025-03-10The end of the output:
Larkfield’s plan for the week of 2025-03-10 is to handle 12,839 orders, worth $596,273, with an expected order cost of $32,038. …
Check: every number is in the plan.As in Module 5, 12,839 is the number of units to order, not customer orders: the number is right, and the word is wrong.
The check flagged 1 of the 10 recorded notes. Priya then read all 10. She found “expected order value” in 4 notes, and “orders” for units in this one. The number check passed all 5. So a person stays in the loop.
When to allow overrides, and what they cost
Section titled “When to allow overrides, and what they cost”Allow overrides when people know things that the data does not show: the weather, a supplier’s rule, a local event. Require a reason, and measure FVA every quarter.
They have costs:
- Time. Each override is a person’s work, every week. If the FVA of a reason stays negative, stop that kind of override.
- Patience. FVA needs 14 finished days and many overrides before it means much. Ten overrides are a first look.
- Honest reasons. “Gut feeling” cannot be measured or learned from. A short, specific reason can.
- A hidden fix. An override that helps every spring is a sign that the method is missing something. Fix the method, then the overrides can stop.
The drafted note costs almost nothing in money. Its real cost is the check and the person who reads it. Never send a drafted note without both.
Common mistake
An override with no reason
Symptom: Kenji adds a row to his own overrides file in a hurry, with the reason column empty, and runs the week with --overrides my-overrides.csv. The job stops before it plans:
my-overrides.csv: every override needs a reasonCause: overrides.read() refuses a row with an empty reason. Without a reason, nobody can later group the overrides, see which kind helps, or learn from them.
Fix: write a short, specific reason, for example local school fair on Saturday, and run again. Agree on a few standard reasons in the team, so that the FVA by reason has enough rows.
Consequence in AI work: a human change to a model’s output without a record is invisible. Later, nobody can say whether the model or the person was wrong, and the same mistake repeats. Record every human change with who and why, next to the model’s own output.
Your turn: one FVA by hand
Section titled “Your turn: one FVA by hand”Take the override of 14 April for P141, a duvet at 74.99 dollars, in the south region. The job ordered 57, Kenji ordered 80, and 58 were sold. Before you look at the fva output again, compute the cost of each order and the FVA. Use the course’s costs: a lost sale costs 35% of the price, a left-over item 25%.
Show the answer
- The job’s order of 57 was 1 short: 1 × 0.35 × 74.99 = 26.25 dollars.
- Kenji’s order of 80 left 22 over: 22 × 0.25 × 74.99 = 412.44 dollars.
- FVA = 26.25 − 412.44 = −386.19 dollars.
The fva output has the same numbers. The supplier’s minimum order cost 386 dollars this time. If the duvet sells slowly, ask the supplier for smaller boxes, or order 80 only every second time.
Knowledge check
Section titled “Knowledge check”Check your understanding
What is forecast value added, and what does a value of −499.94 dollars for the hot-weekend override mean?
Show the answer
Forecast value added (FVA) is the cost of the job’s order minus the cost of the planner’s order. Both costs use what was really sold in the 14 days, with the course’s costs. Positive: the override saved money. For the hot weekend, the job ordered 62 parasols for the coast and Kenji 87, and 38 were sold. The job’s order left 24 over (479.94 dollars), Kenji’s left 49 over (979.88 dollars). So the override cost 499.94 dollars more than the job’s order: it did not help.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The 10 overrides have an FVA of −193 dollars in total. Omar asks: should Kenji stop overriding? Answer with the numbers by reason.
Show the answer
No, not as a whole. The season overrides helped every time: +540 dollars in 4 for the early garden season and +171 for the heaters. They add the yearly season that the job’s method cannot see. The losses come from one weekend weather forecast (−500 dollars) and from the supplier’s minimum order of 80 (−386). The second is the cost of a supplier rule, not of Kenji’s judgment. “Gut feeling” lost 19 dollars in 2 small overrides. So keep the season overrides until the method has a yearly season. Stop overrides for one-weekend events on a 14-day order. Record supplier rules separately, and measure again with more overrides.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The note of 10 March passes the number check. Why must a person still read it before it is sent?
Show the answer
The check finds only numbers that are not in the plan. It does not check what the words around a number say. The note calls 12,839 units “orders”, which is wrong: it is the number of units to order. The note of 24 March, and three others, call the order’s value an “expected order value”. A manager could make a wrong decision from either sentence. So the plan gives the numbers, the LLM drafts the words, the check catches unsupported numbers, and a person reads for meaning.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- An override is a person’s change to the job’s number. The job records the new number, its reason and who, and keeps its own number next to it.
- Forecast value added = cost of the job’s order − cost of the override, against the real sales. Kenji’s 10 overrides: −193 dollars in total, 6 helped and 4 did not.
- By reason, the story is clearer: the season overrides saved money, a one-weekend override and a supplier rule cost money.
- A small LLM can draft the forecast note. The numbers come from the plan, a number check flags the ones that are not in it, and a person reads for meaning.
- Your job now plans, keeps, checks, alarms and records people’s changes: a forecast in production.
You can now run a forecast every week, watch it and learn from the people who change it. But all of it used Larkfield’s synthetic data. Next, you take the whole method to real open data, in the case studies.