Memory, quantization and other methods

What you will be able to do

Know before you train whether a method fits your computer. You estimate the memory of full tuning, LoRA and 4-bit QLoRA, and see what quantization saves and changes. Then you choose a method for three hardware budgets.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

• LoRA and the course’s adapter, from Full and parameter-efficient tuning.
• Bytes per number (float32 4, bfloat16 2) and why training needs more memory than inference, from Inference costs in Deep Learning and Transformers Explained.

What you need on your computer

• Your project at the end of the last lesson: python -m pytest gives 19 passed. If it is broken, copy the snapshot end-of-m02 and do the steps of the last lesson again.
• The shell variable S from the last lesson. In a new terminal, set it again.

A desk for editing a book

You must correct a book on a small desk. The book itself takes space. For every page that you may change, you also need a draft sheet and two sheets of notes. If you may change every page, you need the book plus three times the book. If you may change only a few pages, you need the book plus a few sheets. And if the book is printed smaller, it takes less of the desk.

The problem: will it fit?

Tomás has a new idea: “SmolLM2 is tiny. Let’s tune a 7-billion-parameter model. It knows much more.” Kwame asks the first question: will it fit?

Training with the Adam optimizer in float32 keeps four numbers of 4 bytes for each trained weight. They are the weight, its gradient and Adam’s two running averages. That is 16 bytes per parameter. For SmolLM2, 134,515,008 × 16 bytes = 2.15 GB. For a 7B model, 7,000,000,000 × 16 bytes = 112 GB. The course’s laptop has 16 GB.

LoRA changes the sum

With LoRA, every weight is still in memory, at 4 bytes. But gradients and Adam’s numbers exist only for the adapter: 12 bytes for each of its numbers. For SmolLM2: 134,515,008 × 4 + 460,800 × 12 bytes = 0.54 GB. For a 7B model, the frozen weights alone are 28 GB.

So the frozen weights now take almost all the memory. To save more, store them with fewer bytes.

Quantization: fewer bits per weight

Quantization stores each weight with fewer bits, by rounding it. A price list rounded to the nearest 5 cents is the same idea. Resource trade-offs in LLMs: How They Work and How to Choose One explains it; here is a tiny example with 4 bits.

Take 8 made-up weights. The one furthest from zero is −0.35. Four bits have 16 possible values; this example uses the whole steps from −7 to 7. So one step is 0.35 ÷ 7 = 0.05. Each weight becomes its nearest step:

Weight (float32)	0.12	−0.05	0.31	−0.22	0.02	−0.35	0.18	0.07
Stored step	2	−1	6	−4	0	−7	4	1
Read back (step × 0.05)	0.10	−0.05	0.30	−0.20	0.00	−0.35	0.20	0.05
Error	−0.02	0	−0.01	+0.02	−0.02	0	+0.02	−0.02

The 8 weights took 32 bytes in float32. In 4 bits, they take 4 bytes, plus 4 bytes for the step size. Real 4-bit formats use groups of 64 weights and better steps, but the idea is the same.

The same 8 made-up weights before and after 4-bit rounding. Each weight moves to the nearest multiple of 0.05; no weight moves more than 0.02.

Text description of the chart

Two rows of 8 dots on an axis from −0.4 to 0.4. The float32 row has the original weights: −0.35, −0.22, −0.05, 0.02, 0.07, 0.12, 0.18 and 0.31. The 4-bit row has the rounded values: −0.35, −0.20, −0.05, 0, 0.05, 0.10, 0.20 and 0.30. Every rounded value is a multiple of 0.05, and each dot is at most 0.02 from its original.

QLoRA combines the two ideas: the frozen model in 4 bits, and a LoRA adapter trained at full precision on top. For a 7B model, that is 3.79 GB. It has two costs. The rounding changes the model’s answers a little, so you measure quality again. And the usual 4-bit training library, bitsandbytes, needs a GPU: QLoRA is not an option on the course’s laptop processor.

Estimate the memory in the project

1. Copy the estimator and its test, and run it:

   macOS and Linux

   cp $S/adapt/memory.py adapt/
   cp $S/tests/test_memory.py tests/
   python -m adapt memory

   Windows

   Copy-Item $S\adapt\memory.py adapt\
   Copy-Item $S\tests\test_memory.py tests\
   python -m adapt memory

   The output:

   ESTIMATE: weights + gradients + AdamW states. Activations are not included (measure them).
   model          method                             GB  laptop 16 GB     GPU 24 GB     GPU 80 GB
   SmolLM2-135M   full fine-tuning (float32)       2.15          fits          fits          fits
   SmolLM2-135M   LoRA, base in float32            0.54          fits          fits          fits
   SmolLM2-135M   LoRA, base in bfloat16           0.27          fits          fits          fits
   SmolLM2-135M   QLoRA, base in 4 bits            0.07          fits          fits          fits
   a 1.7B model   full fine-tuning (float32)      27.20            no            no          fits
   a 1.7B model   LoRA, base in float32            6.87          fits          fits          fits
   a 1.7B model   LoRA, base in bfloat16           3.47          fits          fits          fits
   a 1.7B model   QLoRA, base in 4 bits            0.92          fits          fits          fits
   a 7B model     full fine-tuning (float32)     112.00            no            no            no
   a 7B model     LoRA, base in float32           28.29            no            no          fits
   a 7B model     LoRA, base in bfloat16          14.29            no          fits          fits
   a 7B model     QLoRA, base in 4 bits            3.79          fits          fits          fits
   a 70B model    full fine-tuning (float32)    1120.00            no            no            no
   a 70B model    LoRA, base in float32          282.88            no            no            no
   a 70B model    LoRA, base in bfloat16         142.88            no            no            no
   a 70B model    QLoRA, base in 4 bits           37.88            no            no          fits
   fits = the estimate uses at most 70% of the memory (the rest: activations, the system).
   QLoRA needs a GPU library (bitsandbytes); on a laptop CPU it is not an option.

2. Run the tests. The last line says 21 passed.

The sizes other than SmolLM2 are round sizes of common model families, an assumption. The estimator also assumes that LoRA trains 0.34% of any model, as in SmolLM2. Models and libraries change fast: these numbers describe this model and these versions.

• Full tuning
• LoRA
• QLoRA, base in 4 bits

The same 7B model with four methods (estimates, activations not included). 70% of a 16 GB laptop is 11.2 GB; of a 24 GB GPU, 16.8 GB; of an 80 GB GPU, 56 GB.

Text description of the chart

A column chart with four columns, estimates for a 7B model. Full tuning in float32: 112.0 GB. LoRA with the base in float32: 28.3 GB. LoRA with the base in bfloat16: 14.3 GB. QLoRA with the base in 4 bits: 3.8 GB. With 70% of the memory usable, only QLoRA fits the 16 GB laptop, and it needs a GPU. LoRA in bfloat16 fits a 24 GB GPU. Full tuning fits none of the three budgets.

Other methods, other data

Two more methods appear in every discussion about adapting models. Each one needs different data.

Method	What it trains	The data it needs	For Larkfield’s task
Distillation	A small “student” model copies the answers of a large “teacher” model	Many unlabelled tickets, and the teacher’s cost to answer them	Possible later; the student also copies the teacher’s mistakes, and the provider’s terms must allow it
Preference tuning (for example DPO)	The model learns which of two replies people prefer	Pairs of replies, one marked as better	No: it changes how a model writes, and our task has one right answer

Discuss: which computer, which method?

Kwame has three options for the experiment: the laptop (16 GB, no GPU), one rented GPU with 24 GB, or one with 80 GB.

• Tomás: “The 24 GB GPU. QLoRA on a 7B model. A bigger model knows more, French too.”
• Kwame: “Renting costs money and needs an account. The laptop costs nothing extra.”
• Priya: “What must the first run answer?”
• Grace: “Remember that 10 of the 102 dev cases are in French.”

Before you read on, answer these:

1. Which methods fit each option? Use the table above.
2. Which failure could a larger base model reduce? Which could it not?
3. Which evidence would justify the rented GPU?

The course’s answer. The laptop, LoRA rank 8, SmolLM2. The first run must show whether training on our records teaches the format and the routing. That question needs a small, cheap run that anyone can repeat. The laptop has room: the run’s measured peak is 1.88 GB, as the failure case below shows. A larger model may read French better, but it cannot learn rules H2 to H5 or French examples that are not in the data.

What would change the answer: the small adapter may learn English routing but fail on French, because the base model reads French badly. Then QLoRA on a 7B model is the next measured step. It should come with French training records.

Common mistake

The estimate fits, the run does not

Symptom: Tomás reads the table: “LoRA on a 7B model in bfloat16 needs 14.29 GB. The laptop has 16 GB, so it fits.” The estimator says no.

Cause: the estimate counts only the weights, the gradients and Adam’s numbers. It leaves out the activations, which grow with the batch and the text length, and the running program itself. The course’s own run shows the gap. For SmolLM2, the estimate is 0.54 GB, but the training process measured a peak of 1,882 MB.

• Estimate: weights, gradients, Adam
• Measured peak of the training process

The course's LoRA run, batch 8: the estimate is 0.54 GB, the measured peak is 1.88 GB. The difference is activations and the program itself.

Text description of the chart

Two columns. The estimate of the weights, gradients and Adam’s numbers for LoRA on SmolLM2-135M in float32: 0.54 GB. The measured peak memory of the whole training process, at batch 8: 1.88 GB, about three and a half times the estimate.

Fix: treat the estimate as the smallest possible amount, and keep 30% free, as fits does. Then measure a short run before a long one. Module 4 does that.

Guided practice: three budgets

Copy the team’s worked example and read it:

macOS and Linux

cp $S/docs/method-choice.md docs/
cp $S/README.md .

Windows

Copy-Item $S\docs\method-choice.md docs\
Copy-Item $S\README.md .

docs/method-choice.md takes the three budgets of the table, the laptop and the two GPUs. For each one, it lists what fits, the choice and the reason. Each reason names the task, the memory and the data. Your project now equals the snapshot end-of-m03.

Your turn: three other budgets

Choose a method for each budget, and give one reason from the task and one from the memory.

1. An office laptop with 8 GB and no GPU.
2. One GPU with 48 GB.
3. A server with 64 GB and no GPU.

Show a worked solution

Usable memory is 70%: 5.6 GB, 33.6 GB and 44.8 GB.

1. 8 GB laptop: LoRA on SmolLM2 (estimate 0.54 GB). Full tuning (2.15 GB) also fits, but the 900 records argue for an adapter. A 1.7B model in float32 (6.87 GB) does not fit, and QLoRA needs a GPU.
2. 48 GB GPU: LoRA or QLoRA on a 7B model fits (28.29, 14.29 and 3.79 GB). Use it only after the small model’s evaluation shows a failure that a larger model could fix. The 70B model (37.88 GB) does not fit.
3. 64 GB server, no GPU: 7B LoRA in float32 (28.29 GB) fits in memory. But a 7B model has about 52 times the weights of SmolLM2 to compute with at every step, an estimate. On a processor, that is too slow for a first experiment. Choose LoRA on SmolLM2, as in the course.

Completion check: justify a method

A made-up company, Harbour Bikes, wants to sort repair requests into three teams, as strict JSON. It has 400 labelled requests, all in English, and one laptop with 16 GB and no GPU. The requests must not leave its computers. Write five to eight sentences: the method, the model size, the memory estimate, and one result that would change your choice.

Show a worked solution and what reviewers check

“The requests must stay on our computers, so a hosted model is out. First, we compare a prompt on a small local model and a word-count classifier. If we adapt, we use LoRA, rank 8, on a small model such as SmolLM2-135M. The estimate is 0.54 GB, far below 70% of 16 GB. With 400 examples, full tuning has too many weights for too few examples. QLoRA needs a GPU, which we do not have. If both losses stay high, we try a higher rank, then a larger model with LoRA in bfloat16 (1.7B: 3.47 GB).”

Reviewers mark it demonstrated when it does four things. It names the method and the model, and it uses an estimate with headroom. It gives a reason from the data, and it names evidence that would change the choice. It is partially demonstrated when one of these is missing.

Knowledge check

Check your understanding

Why does full tuning in float32 need about 16 bytes per parameter, and why does LoRA need much less?

Show the answer

Full tuning trains every weight. Each one needs 4 bytes for the weight, 4 for its gradient and 8 for Adam’s two running averages: 16 bytes. LoRA freezes the model. The frozen weights still need their 4 bytes each, but gradients and Adam’s numbers exist only for the adapter’s few numbers. For SmolLM2, that is 0.54 GB instead of 2.15 GB.

Check your understanding

What does quantization save, and what can it change? Why is QLoRA not used on the course's laptop?

Show the answer

Quantization stores the weights with fewer bits by rounding them, so the frozen model takes less memory: 4 bits are one eighth of float32. The rounding moves every weight a little, so the model’s answers can change, and quality must be measured again. QLoRA keeps the frozen model in 4 bits and trains a LoRA adapter on top. Its usual library, bitsandbytes, needs a GPU, and the course’s laptop has none.

Check your understanding

The estimate for a run is 9 GB, and your computer has 12 GB. Do you start the long run? Which evidence would make you change your mind?

Show the answer

Not yet. 9 GB is more than 70% of 12 GB (8.4 GB), and the estimate leaves out activations and the program itself. The course’s LoRA run needed 1.88 GB against an estimate of 0.54 GB. Run a few steps first and measure the peak memory. If the measured peak stays well below 12 GB, start the long run. If not, use a smaller batch, shorter texts, a smaller model or more memory.

Recap

• Training memory starts with 16 bytes per trained parameter in float32: SmolLM2 2.15 GB, a 7B model 112 GB. LoRA trains few numbers, so the frozen weights dominate: 0.54 GB for SmolLM2.
• Quantization rounds weights to fewer bits. QLoRA freezes the model in 4 bits under a LoRA adapter: 3.79 GB for a 7B model, with a GPU and a quality check.
• An estimate is the smallest possible amount. The course’s run measured 1.88 GB against 0.54 GB, so keep 30% free and measure.
• Distillation needs a teacher and many tickets; preference tuning needs pairs of replies. Neither fits this task now.
• The module’s choice: LoRA, rank 8, on SmolLM2-135M, on the laptop, because the data, not the hardware, limits what the run can learn.

Next, you pin the model, plan the run and set its configuration.

Next: Plan and configure the run