Plan and configure the run

What you will be able to do

Write down every setting of a training run before it starts, and check what the run will cost in time and memory. Then the run can be repeated, stopped safely and compared with the next one.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• What LoRA trains (rank 8 on the query and value layers: 460,800 parameters, 0.34% of the model), from Full and parameter-efficient tuning.
• The memory estimate for LoRA on this model, from Memory, quantization and other methods.
• Batches, epochs and a validation set, from Batches, epochs and validation in Deep Learning and Transformers Explained.
• The learning rate, from Learning rate and optimizers.

What you need on your computer

• The project ~/projects/ticket-adapter at the end of Module 3: python -m pytest shows 21 passed. If it does not, copy the snapshot end-of-m03 as in Use a snapshot.
• The course’s labs repository in ~/projects/nextia-learning-labs (the same page tells you how to get it).
• About 1.2 GB of free disk space: about 720 MB for PyTorch and the other libraries, and 269 MB for the model.

Tested on macOS (Apple silicon) with Python 3.12.13, PyTorch 2.14.1, transformers 5.19.0 and peft 0.21.2. The Windows and Linux commands are written, not tested.

A long drive, planned

Before a long drive, you plan. The trip is 450 km, and you drive at about 90 km an hour, so it takes about five hours. You stop for fuel every 150 km. Every hour, you look at the map and check that you are still on the right road. You write down the plan, so that a friend can repeat the same trip. And you check that the car can carry everything before you start.

A training run needs the same plan. The number of steps is the distance. The checks of the map are the validation losses. The fuel stops are the checkpoints, where a stopped run can start again. The car’s space is the computer’s memory.

The problem: a run that nobody can repeat

Imagine that Tomás trains an adapter in a notebook. He changes the learning rate twice and stops after “about an hour”. He loads the model by its name only, HuggingFaceTB/SmolLM2-135M-Instruct. The adapter looks good. Two weeks later, Priya asks three questions. Which learning rate made it? How many steps? Which files of the base model? Nobody knows.

The last question is the worst. A model name without a revision means “the newest files”. If the model’s owners publish new weights or a new tokenizer, the same command gets other files, and no error tells you. A tokenizer turns text into numbered tokens; if it changes, the adapter sees other numbers for the same words. So the result cannot be repeated, and you cannot compare it with the next run either.

The idea: one file for the run, a plan and pinned versions

The fix has three parts.

One configuration file holds every setting that changes the result. A setting that you choose before training, and that the model does not learn, is a hyperparameter. The course’s run is configs/lora-r8.toml:

configs/lora-r8.toml

name = "lora-r8"
seed = 12
threads = 2                 # CPU threads: the run must leave the rest of the computer usable

[data]
train = "data/prepared/train.jsonl"   # from: python -m adapt prepare (900 records)
val = "data/prepared/val.jsonl"       # 100 records, the same teams, never trained on
max_tokens = 320            # longer examples lose the end of the ticket text, never the answer

[lora]
rank = 8
alpha = 16                  # the update is scaled by alpha / rank = 2
dropout = 0.05
targets = ["q_proj", "v_proj"]

[train]
learning_rate = 2e-4
batch_size = 8
epochs = 2
eval_every = 25             # steps between validation losses
checkpoint_every = 50       # steps between checkpoints (a stopped run loses at most 50 steps)
gradient_checkpointing = true   # less memory, about 20% more time
keep_checkpoints = 2

You know most names from Module 3 and from the deep learning course. Four are new here:

• max_tokens is the sequence length: the most tokens that one training example may have. A longer ticket loses the end of its text, never the answer. A test in the project checks this.
• checkpoint_every: a checkpoint is a saved copy of the training state, from which a stopped run continues. You use it in Monitor and recover.
• seed fixes the random choices, such as the order of the batches. The same seed gives the same run on the same computer.
• threads limits the run to 2 of the processor’s cores, so the computer stays usable.

A plan turns the file into numbers before the run starts. Here is the arithmetic, with every value visible. 900 training records in batches of 8 make 112.5 batches, so 113 steps per epoch: the last batch has only 4 records. Two epochs make 226 steps. A validation loss every 25 steps gives 9 of them (25, 50, …, 225), and a checkpoint every 50 steps gives 4 (50, 100, 150, 200). The run also validates and saves at its last step, 226.

The plan of the course's run: 226 steps, a validation loss every 25 steps and a checkpoint every 50. A stop loses at most the steps since the last checkpoint.

Text description of the diagram

The top row is the arithmetic. 900 training records, divided by a batch size of 8, give 113 steps per epoch. Two epochs give 226 steps in the run. Below it is a timeline from step 0 to step 226. Its first half is epoch 1 (steps 1 to 113) and its second half is epoch 2 (steps 114 to 226). Above the timeline, a blue dot marks each validation loss: at steps 25, 50, 75, 100, 125, 150, 175, 200, 225 and 226. Below the timeline, a dark square marks each checkpoint: at steps 50, 100, 150, 200 and 226. A red dashed line at step 120 shows a stop. The run resumes from the checkpoint at step 100, so steps 101 to 120 run again.

Pinned versions fix every file and library. adapt/model.py names the model and its exact revision. The tokenizer files in model/ come from the same revision, and download refuses weights whose config.json differs from model/config.json. requirements.txt lists an exact version of each library, such as torch==2.14.1 and peft==0.21.2.

adapt/model.py (part)

MODEL_ID = "HuggingFaceTB/SmolLM2-135M-Instruct"
REVISION = "12fd25f77366fa6b3b4b768ec3050bf629380bac"

Time and memory, estimated and then measured

Memory, quantization and other methods estimated 0.54 GB for LoRA on this model: the weights, the gradients and the optimizer’s state. The estimate leaves out the activations, the intermediate results of each layer that training keeps for the backward pass. So we measured the whole process. The course’s run reached a peak of 1,882 MB.

Gradient checkpointing saves memory. It keeps only some of the activations and computes the others again during the backward pass. So it costs time. In a short measurement of 8 steps at batch 4, the steps took 5.5 s with it and 4.3 s without it. (The run’s own checkpoints, the saved files, are a different thing.)

Measured peak memory on the course's computer. Training at batch 8 needs 1,882 MB. Without gradient checkpointing, even batch 4 needs 2,472 MB.

Text description of the chart

The chart shows four measured peaks of memory, for the whole Python process. Answering tickets with the adapter loaded needs 1,262 MB. Training with gradient checkpointing on needs 1,724 MB at batch 4 (8 steps) and 1,882 MB at batch 8 (the course’s run). Training at batch 4 with gradient checkpointing off needs 2,472 MB, more than the course’s run at batch 8. Every run stayed under the course’s limit of 3,000 MB.

The course’s run took 367 seconds (about 6 minutes) on an Apple M4 with 2 threads. Times depend on your computer and on what else runs on it; memory and results do not. These numbers are a snapshot: libraries and models change fast.

When the plan says no. Do not train if your computer has less than about 4 GB of free memory, or no room for 1.2 GB of files. The course’s recorded runs are in the project, in runs/. Every later lesson works with them, without training.

Discuss: how big should the first run be?

Monday morning. The team agrees to try one adapter. Now they must choose its settings.

• Tomás: “Let’s go big. Rank 64, all seven layers, ten epochs. It runs overnight. Why train a weak adapter first?”
• Kwame: “This laptop also runs other jobs. I want a run under 30 minutes and 3 GB. If it stops, we must not lose the night.”
• Priya: “The first run should be small. It gives us a full curve quickly. Then we change one thing and compare.”

Before you read on, answer these:

1. Which run would you start first, and why?
2. What would you need to see before you train longer?
3. If a setting is wrong, what does each choice cost?

The course’s answer. Start with Priya’s run: rank 8 on two layers, 2 epochs. It took about 6 minutes and 1,882 MB, so a mistake costs minutes, not a night. Tomás’s run changes many things at once. If it works, nobody knows which change helped; if it fails, nobody knows which change hurt. It also has no measurement. Rank 64 on all seven layers trains 19,537,920 parameters, 42 times as many (python -m adapt params --rank 64 with all seven --targets). Its memory and time are guesses. A small run first, then one change at a time, gives evidence for each choice. In the next lesson, the curve of this first run decides the second run.

What would change the answer: a computer with a graphics card and a measured budget, or a task that is already well understood. Then a larger first run can be reasonable, but still with one configuration file and a plan.

Guided practice: set up the run

Work in the project folder, with the virtual environment active.

1. Get the newest labs files: git -C ../nextia-learning-labs pull.

2. Copy this lesson’s files from the snapshot end-of-m04:

   macOS and Linux

   S=../nextia-learning-labs/C12/snapshots/end-of-m04
   mkdir configs
   cp $S/requirements.txt $S/pytest.ini .
   cp $S/adapt/__main__.py $S/adapt/model.py $S/adapt/config.py adapt/
   cp $S/configs/lora-r8.toml configs/

   Windows

   $S = "..\nextia-learning-labs\C12\snapshots\end-of-m04"
   mkdir configs
   Copy-Item $S\requirements.txt, $S\pytest.ini .
   Copy-Item $S\adapt\__main__.py, $S\adapt\model.py, $S\adapt\config.py adapt\
   Copy-Item $S\configs\lora-r8.toml configs\

3. Install the libraries. On Windows and Linux, install PyTorch first from its CPU-only index. Without it, pip on Linux gets the much larger version for graphics cards.

   macOS and Linux

   # Linux only, first: python -m pip install torch==2.14.1 --index-url https://download.pytorch.org/whl/cpu
   python -m pip install -r requirements-dev.txt

   Windows

   python -m pip install torch==2.14.1 --index-url https://download.pytorch.org/whl/cpu
   python -m pip install -r requirements-dev.txt

   It takes a few minutes. The environment grows to about 720 MB.

4. Download the model’s weights: python -m adapt download. The last line shows the revision, the size and the folder:

   HuggingFaceTB/SmolLM2-135M-Instruct at revision 12fd25f77366: 269 MB in <your cache>/hub/models--HuggingFaceTB--SmolLM2-135M-Instruct/snapshots/12fd25f77366fa6b3b4b768ec3050bf629380bac

   Your cache folder is different. A warning about “unauthenticated requests” is normal: no account is needed.

5. Plan the run: python -m adapt plan configs/lora-r8.toml. Compare each number with the arithmetic above:

   lora-r8: 900 training records, 100 validation records
     batch 8, 2 epochs = 113 steps per epoch, 226 steps (1808 examples seen)
     validation loss every 25 steps (9 times), checkpoint every 50 steps (4)
     LoRA rank 8, alpha 16, dropout 0.05, on q_proj, v_proj; learning rate 0.0002; at most 320 tokens per example; 2 threads

   The plan counts 226 × 8 = 1,808 examples. The last batch of each epoch has only 4 records, so the run really sees 1,800.

6. Run the tests: python -m pytest. The last line says 21 passed. The tests of the training code arrive in the next lesson.

Your turn: a smaller batch

Copy configs/lora-r8.toml to configs/batch4.toml. In the copy, change name to "batch4" and batch_size to 4. Before you run python -m adapt plan configs/batch4.toml, predict the steps per epoch, the steps, the validations and the checkpoints. Then run it.

Show the answer

batch4: 900 training records, 100 validation records
  batch 4, 2 epochs = 225 steps per epoch, 450 steps (1800 examples seen)
  validation loss every 25 steps (18 times), checkpoint every 50 steps (9)
  LoRA rank 8, alpha 16, dropout 0.05, on q_proj, v_proj; learning rate 0.0002; at most 320 tokens per example; 2 threads

900 ÷ 4 = 225 steps per epoch exactly, so the plan’s 1,800 examples are now right. The run has twice as many steps, so it validates and saves twice as often on the same data. It is not faster: it sees the same 1,800 examples, in smaller pieces. It also makes twice as many updates with the same learning rate, so its result can differ. Change one thing, and measure. Delete configs/batch4.toml when you finish: the next lessons do not use it.

Common mistake

A typing mistake in the learning rate

Symptom: Sam copies the configuration and types learning_rate = 2e4 (the minus sign is missing). python -m adapt plan configs/typo.toml stops with a traceback. Its last line:

ValueError: configs/typo.toml: learning_rate must be between 0 and 1

Cause: 2e4 is 20,000, not 0.0002. load_config() in adapt/config.py checks the values before anything runs. It refuses a learning rate outside 0 to 1, and fewer than 64 tokens per example. It also refuses a rank, batch size or number of epochs below 1.

Fix: write learning_rate = 2e-4, and run plan again.

Consequence in AI work: without the check, the run starts, and the loss jumps or becomes “nan” (not a number) after a few steps. That costs minutes here and hours on a large model. A check that refuses impossible settings before the run starts is cheap. It does not catch a value that is possible but bad, such as 2e-3: only the curves of the next lesson show that.

See it end to endShould this team fine-tune?Case study

The case study runs this recipe unchanged on new data: 813 records, 204 steps, about 4 minutes on a laptop CPU and 1,702 MB at the peak. Then it makes one change, more data, and keeps the targets as they were written.

Knowledge check

Check your understanding

A configuration has 900 training records, batch_size 6, epochs 3, eval_every 25 and checkpoint_every 50. How many steps does the run have, and at which steps does it save a checkpoint?

Show the answer

900 ÷ 6 = 150 steps per epoch, and 3 epochs give 450 steps. It saves a checkpoint every 50 steps (50, 100, …, 450): 9 checkpoints. Step 450 is also the last step, so no extra save is needed. It validates every 25 steps: 18 times.

Check your understanding

Tomás says: "The model name is enough; everyone knows which model we used." Which two things can go wrong without the revision, and how does this project protect against them?

Show the answer

Without a revision, the name means the newest files. New weights would give the adapter another base, and a new tokenizer would give other token numbers for the same words. Neither change raises an error. The project pins the exact revision in adapt/model.py and keeps the tokenizer files of that revision in model/. download refuses weights whose config.json differs, and requirements.txt pins each library’s version.

Check your understanding

Which evidence would make you agree to Tomás's large first run (rank 64 on all layers, ten epochs, overnight)?

Show the answer

Measured evidence, not hope. First, its peak memory and time, measured on the computer that runs it, below the limits (here 3 GB and 30 minutes). Second, a reason to expect that the small run is not enough. An example: a small run whose validation loss stopped falling while the results stayed too low. Even then, change one thing at a time, so that each result shows which change made the difference.

Recap

• A configuration file holds every setting that changes the result: data, LoRA, learning rate, batch size, epochs, sequence length, seed, threads, validation and checkpoints. The file is the record of the run.
• plan turns it into numbers before the run. 900 records in batches of 8 give 113 steps per epoch and 226 steps. The run has 9 validations and 4 checkpoints, plus one of each at the last step.
• Pin the revision of the model and its tokenizer, and the version of each library. A name alone means “the newest files”, and a change raises no error.
• An estimate leaves things out. The run’s measured peak was 1,882 MB; gradient checkpointing saves memory (2,472 MB without it at batch 4) and costs time.
• The first run is small and quick. The next run changes one thing, chosen from the evidence.

Next, you train the run, read its two curves, and stop and resume it from a checkpoint.

Next: Monitor and recover