Lifecycle planning

What you will be able to do

“Do not ship” is a result, and it needs a plan too. You write down what the team keeps, what would make it try again, what must exist before any deployment, and which licences to review.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

• The bundle, the fixed checks and the rollback, from Package and deploy an adapter.
• The probabilities behind the missed safety tickets, from Regressions and operational value.

What you need on your computer

• ~/projects/ticket-adapter after the previous lesson, with .venv active: python -m pytest ends with 36 passed, 3 deselected, and bundle/ exists. If it does not, do the steps of Make the bundle again.

Tested with Python 3.12.13 on macOS (Apple silicon). The Windows commands are written, not tested.

A failed driving test

You fail a driving test. The examiner’s sheet lists two faults, both at junctions. You keep the sheet. You do not practise parking, which you already pass. You practise junctions, then book a new test on a route that you have not driven.

The problem: a “no” that leaves nothing behind

Imagine that the team deletes runs/ and bundle/ after the result memo. Months later, Tomás proposes a third run: “12 epochs. The loss will be even lower.” Without the record, a lower loss looks like progress.

The record says otherwise. The second run changed only the epochs, from 2 to 6:

Run	Validation loss	Team (dev)	Needs a person, caught (dev)
lora-r8, 2 epochs	0.0584	81 of 100	0 of 23
lora-r8-long, 6 epochs	0.0193	89 of 100	0 of 23

More training on the same data improved the routing, and did not change the safety decision.

The idea: a lifecycle plan

A lifecycle plan says what happens to a model after the decision, in four parts:

1. Keep the evidence: the bundle, the configurations, the data manifest and the rejected records, the saved replies and the memos.
2. Retraining triggers: events, written down in advance, that start a new attempt.
3. Before any deployment: monitoring, corrections, rollback and the compatibility check.
4. Licence reviews: when to check the terms of the base model, the data and the libraries again.

Monitoring means counting what a deployed model does, every week. Data corrections are wrong labels that a review finds, fixed in a recorded way.

The adapter's lifecycle: the path the team took after the decision, and the path that only a passing adapter would take.

Text description of the diagram

The adapter was trained, compared on dev and then on the holdout once, then bundled and verified. The decision was “do not ship”.

On the path that the team took, it keeps the evidence. It watches three triggers: reviewed examples of rules H1 to H5 and of French tickets; a change in the hosted model’s price or terms; a new base model with a permissive licence. When a trigger fires, a new attempt starts: prepare, one run, compare on dev, decide, and read a new holdout version once.

The path not taken needs a passing adapter. Then: verify at every start, a weekly review that counts missed escalations first, a corrections file for wrong labels, the hosted assistant as the rollback, and a licence review at each new version.

A tiny example: which trigger is real?

A trigger must point at the cause. Count the tickets that need a person, rule by rule, and the French tickets, in the training records and in the evaluation cases:

• Training records (900)
• Evaluation cases, dev and holdout (200)

H1 to H5: tickets that need a person, by rule. FR: French tickets. Training covers only rule H1; the evaluation cases test all five rules and French.

Text description of the chart

A column chart with two columns per rule: training records, then evaluation cases (dev and holdout together). H1: 78 and 23; H2: 0 and 5; H3: 0 and 6; H4: 0 and 9; H5: 0 and 4; French tickets: 0 and 20.

What this tells us. The adapter saw 78 examples of rule H1 and none of the other rules. The probabilities from Module 5 match. Its probability of true was 0.01 to 0.38 on the 10 dev cases under H1. On the 13 under H2 to H5, it was 0.000 to 0.024. It learned a little of the rule it saw, and nothing of the others. So the real trigger is reviewed examples of every rule and of French tickets, not more epochs.

Add the plan to the project

1. Set S as in the previous lesson, if you opened a new terminal.

2. Copy the plan and the course-end README:

   macOS and Linux

   cp $S/docs/lifecycle-plan.md docs/
   cp $S/README.md .

   Windows

   Copy-Item $S\docs\lifecycle-plan.md docs\
   Copy-Item $S\README.md .

3. Run python -m pytest. The last line still starts with 36 passed, 3 deselected.

4. On macOS and Linux, compare your project with the snapshot:

   diff -r -x .venv -x __pycache__ -x .pytest_cache -x .ruff_cache -x bundle $S .

   It prints nothing, except an Only in . line for each file that you wrote yourself, such as docs/my-result-memo.md. Your project is now the course-end project.

5. Read docs/lifecycle-plan.md. Find why the base model is not a rollback.

Discuss: what should the next attempt be?

Months later, the team meets again.

• Tomás: “A third run: 12 epochs and rank 16.”
• Grace: “My team can write reviewed tickets for rules H2 to H5, and in French.”
• Kwame: “Is any attempt worth it? The hosted assistant costs about US$8.53 a month (at 822 tickets a day, an assumption).”

Before you read on, answer these:

1. Which proposal is a retraining trigger in the plan?
2. Is Kwame right that the attempt is not worth it?
3. The holdout cases were read once, for the result memo. Can the next attempt use them again?

The course’s answer. Grace’s proposal is trigger 1; Tomás’s is not. His run changes the settings on the same data, and the record shows what that does. The next attempt starts with prepare: the coverage must show no zeros before anyone trains. Kwame is right that money alone does not pay for an attempt. But keeping tickets on Larkfield’s computers matters if the hosted model’s terms change: trigger 2. The holdout was read and discussed, so it is no longer unseen. Grace’s team writes a new holdout version first, as Production feedback in Evaluating and Testing AI Systems explains.

What would change the answer: a change of the hosted model’s terms that keeps Larkfield’s tickets off it (trigger 2). Then the next attempt is urgent, with the same targets and a new holdout version.

Common mistake

A label corrected by hand

Symptom: a reviewer wants to change one label. To see what happens, make a constructed edit: in data/source/needs_human.csv, change the line of T-60002 to T-60002,true,H2,read: Grace changed it. Run python -m pytest:

FAILED tests/test_prepare.py::test_the_prepared_data_is_the_same_every_time
FAILED tests/test_records.py::test_every_data_file_matches_its_recorded_source
FAILED tests/test_records.py::test_89_training_tickets_need_a_person_and_each_label_says_how_it_was_made
3 failed, 33 passed, 3 deselected, 1 warning in 27.48s

Cause: data/sources.toml records the SHA-256 of every data file, as in Example structure and permissions. The file changed, and nothing records who changed it or why.

Fix: restore the file (cp $S/data/source/needs_human.csv data/source/, or Copy-Item on Windows), and the tests pass again. Write each correction in a new file, with the ticket, the new label, the reviewer and the reason. List it in data/sources.toml, and make the next prepare read it. The course’s project does not have that step yet.

Module practice: deliver a reproducible bundle

Worked example. A bundle is reproducible when the same run gives the same files. Make a second bundle and compare the SHA-256 of each file:

macOS and Linux

python -m adapt bundle lora-r8-long --out bundle2
shasum -a 256 bundle/bundle.json bundle2/bundle.json bundle/MODEL_CARD.md bundle2/MODEL_CARD.md

Windows

python -m adapt bundle lora-r8-long --out bundle2
Get-FileHash bundle\bundle.json, bundle2\bundle.json, bundle\MODEL_CARD.md, bundle2\MODEL_CARD.md

On macOS:

b642f595481e87169742ba1dd5591520be701a179ce65849e3cf51d8a93a8dad  bundle/bundle.json
b642f595481e87169742ba1dd5591520be701a179ce65849e3cf51d8a93a8dad  bundle2/bundle.json
7e7403d66c80a17ba09611cd1f3b9bb5524855fc7e305590f79f2ce355da2167  bundle/MODEL_CARD.md
7e7403d66c80a17ba09611cd1f3b9bb5524855fc7e305590f79f2ce355da2167  bundle2/MODEL_CARD.md

The pairs are identical: bundle.json holds no date and no local path. Delete bundle2 afterwards.

Your turn. Package the first run, lora-r8, with its own saved replies, and verify it:

python -m adapt bundle lora-r8 --system adapter --out bundle-r8
python -m adapt verify bundle-r8

Predict first: which checks fail? Then compare the two model cards.

Show an example answer

lora-r8 on HuggingFaceTB/SmolLM2-135M-Instruct@12fd25f77366: base + adapter, loaded at runtime
  pass  C-01  expected delivery/false  got '{"team": "delivery", "needs_human": false}'
  pass  C-02  expected delivery/false  got '{"team": "delivery", "needs_human": false}'
  pass  C-03  expected returns/false  got '{"team": "returns", "needs_human": false}'
  pass  C-04  expected payment/false  got '{"team": "payment", "needs_human": false}'
  pass  C-05  expected warranty/false  got '{"team": "warranty", "needs_human": false}'
  pass  C-06  expected account/false  got '{"team": "account", "needs_human": false}'
  FAIL  C-07  expected warranty/true  got '{"team": "warranty", "needs_human": false}'
  FAIL  C-08  expected warranty/true  got '{"team": "warranty", "needs_human": false}'
6 of 8 checks passed

The same 6 of 8 and the same two failures. The model cards differ: on dev, run 1 has team 81/100 and decisions 68/102, run 2 has 89/100 and 74/102. Both caught 0 of 23 on dev and 0 of 24 on holdout.

Module check: load and verify a fixed set of behaviours

1. In a fresh copy of the snapshot end-of-m06, with a new .venv and the downloaded model, run python -m adapt bundle lora-r8-long and python -m adapt verify bundle. Say what the exit code means for a deployment.
2. Add two checks to a copy of the bundle, not to data/checks.jsonl: a legal threat (rule H2) and a French safety ticket (rule H1). Predict each result, then run verify on the copy.
3. Write three lines for the team: what passed, what failed, and what the plan says next.

Show what a complete answer contains

1. 6 of 8 checks passed, exit code 1 (C-07, C-08 fail). A deployment script stops on it, so this bundle is never deployed.

2. On macOS and Linux:

   cp -R bundle bundle-extra

   Then add two lines to bundle-extra/checks.jsonl (these two were written for this check):

   {"id": "C-09", "text": "This is the second lawn mower that stopped working after a month. If you do not replace it this week, I will take this to the consumer protection office.", "team": "warranty", "needs_human": true}
   {"id": "C-10", "text": "Le radiateur électrique a fait des étincelles et la prise est devenue très chaude. J'ai peur de l'utiliser.", "team": "warranty", "needs_human": true}

   python -m adapt verify bundle-extra gives:

   lora-r8-long on HuggingFaceTB/SmolLM2-135M-Instruct@12fd25f77366: base + adapter, loaded at runtime
     pass  C-01  expected delivery/false  got '{"team": "delivery", "needs_human": false}'
     pass  C-02  expected delivery/false  got '{"team": "delivery", "needs_human": false}'
     pass  C-03  expected returns/false  got '{"team": "returns", "needs_human": false}'
     pass  C-04  expected payment/false  got '{"team": "payment", "needs_human": false}'
     pass  C-05  expected warranty/false  got '{"team": "warranty", "needs_human": false}'
     pass  C-06  expected account/false  got '{"team": "account", "needs_human": false}'
     FAIL  C-07  expected warranty/true  got '{"team": "warranty", "needs_human": false}'
     FAIL  C-08  expected warranty/true  got '{"team": "warranty", "needs_human": false}'
     FAIL  C-09  expected warranty/true  got '{"team": "warranty", "needs_human": false}'
     FAIL  C-10  expected warranty/true  got '{"team": "delivery", "needs_human": false}'
   6 of 10 checks passed

   Both fail, as the coverage predicts: no H2 example and no French ticket in training. C-10 also goes to the wrong team.

3. For example: “It routes the 6 ordinary checks. It sends no safety or legal ticket to a person, in English or French. Plan: keep the bundle; try again only with reviewed examples of every rule.”

Knowledge check

Check your understanding

The second run had a much lower validation loss than the first. Why is 'train longer' not a retraining trigger in the plan?

Show the answer

The second run trained three times longer on the same data. Validation loss fell from 0.0584 to 0.0193 and routing improved, but it still caught 0 of 23. The cause is the data: 78 examples of rule H1, none of H2 to H5 or French. A trigger must fix the cause.

Check your understanding

The adapter does not ship. Name three things the team keeps, and say why each one matters for the next attempt.

Show the answer

For example: the bundle, with the exact adapter, base revision and evaluation, for a fair comparison later. The data manifest and rejected records: what the team used, and why it removed records. The saved replies: the comparison runs again without a model. The memos: why the team decided.

Check your understanding

Grace's team writes new reviewed tickets for rules H2 to H5. Which evidence, in which order, would you need before you deploy a new adapter?

Show the answer

First prepare: examples of every rule and of French tickets, with provenance. Then one run, compared on dev with the hosted assistant: every target met, all tickets that need a person caught. Then one reading of a new holdout version. Then the bundle, every fixed check passed, the hosted assistant as rollback, and monitoring in place.

See it end to endShould this team fine-tune?Case study

The whole decision, end to end, on a public dataset.

Recap

• A lifecycle plan has four parts: keep the evidence, write the retraining triggers, list what must exist before any deployment, and plan licence reviews.
• A trigger must point at the cause. Here the cause is the data: 78 training examples of rule H1, none of H2 to H5 or French. More epochs did not change the safety result.
• Monitoring, data corrections and a rollback come before a deployment.
• Your project matches the snapshot end-of-m06, the end of the course’s project: 36 passed, 3 deselected.

Models change fast, and this course’s numbers come from one set of runs. The method lasts. Find what fails, and train on data that covers it. Compare on the same held-out cases, and plan what happens after the decision.

Next: Should this team fine-tune?