Full and parameter-efficient tuning

What you will be able to do

Decide which parts of a model training may change. Full tuning changes every weight; a LoRA adapter leaves the model frozen and trains a small extra part. You count both in the course’s real model.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

• What fine-tuning is: more training of a pretrained model on your examples, from Fine-tune a small transformer in Deep Learning and Transformers Explained.
• A weight matrix and its shape, from Tensors and shapes, and the attention layers of a block, from Transformer blocks.
• Larkfield’s training data: 900 records, from Clean, split and check for leakage.

What you need on your computer

• Your project ~/projects/ticket-adapter at the end of Module 2, with its environment active: python -m pytest gives 17 passed. If it is broken, copy the snapshot end-of-m02, as the snapshots page explains.
• The labs repository in ~/projects/nextia-learning-labs, next to your project. Get its newest files with git -C ../nextia-learning-labs pull.

This module needs no PyTorch and no download. The counts come from the model’s configuration file. Tested on macOS with Python 3.12; the Windows commands are written, not tested.

A thick manual and a booklet of notes

A new support agent gets the company’s thick manual. It is good, but it does not know Larkfield’s teams. You can fix that in two ways. You can reprint the whole manual with changes on every page. Or you can keep the manual as it is and clip a thin booklet of notes to it. The agent reads both together.

The booklet has three advantages. It is cheap to write and to copy. You can write a second booklet for another job and keep the same manual. And if a booklet gives bad advice, you take it off, and the original manual is back.

The problem: full tuning changes every weight

Full fine-tuning is the reprint. Training may change every weight of the model. You did this with DistilBERT in the deep learning course. For SmolLM2-135M, the course’s model, that is 134,515,008 numbers.

Each changed version is a new copy of the whole model: 538 MB in float32. Training needs the weights, their gradients and the optimizer’s two numbers per weight: 2,152 MB, as Inference costs measured. And Larkfield has only 900 training records. With so few examples and so many weights, training can change skills that the task never asked for. This is catastrophic forgetting: a model trained on a narrow task loses abilities that it had before.

The idea: freeze the model, train a small adapter

An adapter is the booklet. The model’s weights are frozen: training does not change them. Next to some weight matrices, training adds a few new numbers and changes only those. Methods that work like this are called parameter-efficient fine-tuning, or PEFT.

The most common adapter is LoRA, low-rank adaptation. Training a weight matrix means learning a change to add to it. LoRA does not learn that change number by number. It builds the change from two thin matrices, A and B, and learns only them.

A tiny example: 16 changes from 8 numbers

Take a 4 × 4 weight matrix: 16 numbers. Full tuning learns a change for each of them. LoRA with rank 1 learns one column B of 4 numbers and one row A of 4 numbers. The change is B times A: each row is A multiplied by one number of B.

	A: 0.1	0	−0.2	0.3
B: 1	0.1	0	−0.2	0.3
B: 2	0.2	0	−0.4	0.6
B: 0	0	0	0	0
B: −1	−0.1	0	0.2	−0.3

Eight learned numbers give 16 changes. The rank is the width of the thin matrices. With rank 8, B has 8 columns and A has 8 rows, so the change can go in 8 independent directions. That is the “low rank” in the name.

For a matrix with out rows and in columns, LoRA trains rank × (in + out) numbers instead of out × in. SmolLM2’s query layer, q_proj, is 576 × 576: 331,776 weights. Rank 8 trains 8 × (576 + 576) = 9,216 numbers, 36 times fewer. At the start of training, B is all zeros, so the adapted model answers exactly like the base model.

Where the adapters go

The course puts adapters on two layers of the attention in each of the 30 blocks: the query layer q_proj and the value layer v_proj.

The frozen model with its trainable adapters. In each block, only the thin blue matrices A and B change in training.

Text description of the diagram

On the left, the model is a stack: the embeddings, blocks 1 to 30, then the next-token scores. Every part is dark, because it is frozen. Each block has a small blue LoRA tag. On the right, one block is enlarged; all 30 are the same. In the attention, q_proj is a dark square W of 576 × 576 weights. Next to it are a plus sign, a thin blue column B (576 × 8), a times sign and a thin blue row A (8 × 576). B and A train 9,216 numbers. k_proj (192 × 576) is frozen. v_proj (192 × 576) has its own B (192 × 8) and A (8 × 576): 6,144 trained numbers. o_proj (576 × 576) is frozen, and so are the three feed-forward layers. One block trains 15,360 numbers; 30 blocks train 460,800 of the model’s 134,515,008, or 0.34%.

Count it in the real model

The project counts these numbers from model/config.json alone. That file lists the model’s sizes, so no download is needed.

1. Set a variable for the snapshot, and copy the model’s small files, the counter and its test:

   macOS and Linux

   S=../nextia-learning-labs/C12/snapshots/end-of-m03
   cp -R $S/model .
   cp $S/adapt/params.py adapt/
   cp $S/tests/test_params.py tests/

   Windows

   $S = "..\nextia-learning-labs\C12\snapshots\end-of-m03"
   Copy-Item -Recurse $S\model .
   Copy-Item $S\adapt\params.py adapt\
   Copy-Item $S\tests\test_params.py tests\

2. Run the command:

   python -m adapt params

   The output is an error:

   usage: python -m adapt [-h]
                          {cases,show,systems,failures,score,targets,sources,record,prepare}
                          ...
   python -m adapt: error: argument command: invalid choice: 'params' (choose from cases, show, systems, failures, score, targets, sources, record, prepare)

   The code is there, but the command line does not know the new command yet.

3. Copy the new command line, and run the command again:

   macOS and Linux

   cp $S/adapt/__main__.py adapt/
   python -m adapt params

   Windows

   Copy-Item $S\adapt\__main__.py adapt\
   python -m adapt params

   The output:

   SmolLM2-135M-Instruct: 30 blocks, hidden size 576
   Linear layers in one block (out x in): q_proj 576x576, k_proj 192x576, v_proj 192x576, o_proj 576x576, gate_proj 1536x576, up_proj 1536x576, down_proj 576x1536
   method                                        trainable    share  file (MB)
   full fine-tuning                            134,515,008  100.00%      538.1
   LoRA rank 4 on q_proj, v_proj                   230,400    0.17%        0.9
   LoRA rank 8 on q_proj, v_proj                   460,800    0.34%        1.8
   LoRA rank 16 on q_proj, v_proj                  921,600    0.69%        3.7
   LoRA rank 64 on q_proj, v_proj                3,686,400    2.74%       14.7
   LoRA rank 8 on all 7 linear layers            2,442,240    1.82%        9.8

4. Check it with the tests:

   python -m pytest

   The last line says 19 passed. One test checks that 460,800 is the number that the training library peft prints for the same adapter.

Read the course’s row: rank 8 on q_proj and v_proj trains 460,800 numbers, 0.34% of the model. In float32, that is a file of 1.8 MB instead of 538.1 MB. The real adapter that Module 4 trains is 1,858,776 bytes: these numbers plus a small header.

Explore

What a LoRA adapter trains in SmolLM2-135M

Trainable numbers

460,800

Share of the model

0.34% of 134,515,008

Adapter file (float32)

1.8 MB

Compared with full tuning

292 times smaller than full tuning (538.1 MB)

Rank 8 on q_proj and v_proj: the adapter trains 460,800 numbers, a file of 1.8 MB. The 134,515,008 weights of the model stay frozen. This is the course's choice: rank 8 on q_proj and v_proj.

LoRA rank 8 on q_proj and v_proj: shapes are out × in, numbers for one block

Layer	Frozen W	A	B	Trained
q_proj	576 × 576	8 × 576	576 × 8	9,216
v_proj	192 × 576	8 × 576	192 × 8	6,144
One block				15,360
× 30 blocks				460,800

Each targeted layer gets A (rank × in) and B (out × rank) in each of the 30 blocks. The counts equal python -m adapt params.

Text description of the explorer

The explorer has a slider for the rank, from 1 to 64. Two buttons choose the layers: q_proj and v_proj, or all 7 linear layers. It shows the trainable numbers, their share of the model’s 134,515,008 and the file size in float32. A table shows A and B for each layer of one block. At rank 8 on q_proj and v_proj: 460,800 numbers, 0.34%, 1.8 MB. At rank 16: 921,600 numbers and 3.7 MB, exactly twice as many. At rank 64: 3,686,400 and 14.7 MB. Rank 8 on all 7 layers trains 2,442,240 numbers, 9.8 MB, because the feed-forward layers are wider. The frozen model never changes.

The count grows in step with the rank, because every A and B gets wider. The frozen model stays the same.

When to use LoRA, and what it costs

Use an adapter when you have few examples, little memory, or several tasks on one base model. It is also easy to undo: take the adapter off, and the base model answers again. LoRA is the usual first choice for a narrow task.

It has costs too. An adapter has less room to change the model, so a large change of behaviour may need a higher rank or more layers. It works only with the exact base model that it was trained on. Module 6 shows how to record that. And it still needs the whole model in memory to train, which the next lesson counts.

Discuss: all the weights, or a small adapter?

Kwame asks the team to agree on a method before Module 4’s run.

• Tomás: “LoRA trains 0.34% of the model. That is a toy. Train all the weights, and it will learn our policies.”
• Kwame: “Full tuning is a 538 MB copy for every version. It fits on this laptop, but every retrain makes another one.”
• Priya: “What would the extra weights learn from? 78 of the 900 records need a person, all of them under rule H1, and none is in French.”
• Grace: “Customers do not see the method. They see whether a safety ticket reaches a person.”

Before you read on, answer these:

1. Does full tuning of SmolLM2 fit on the course’s laptop? Is memory the deciding reason here?
2. What could rank 64 learn that rank 8 cannot? What could neither of them learn from these 900 records?
3. Which result of the training run would make you raise the rank?

The course’s answer. LoRA, rank 8, on q_proj and v_proj. Memory is not the deciding reason, because the estimate for full tuning, 2.15 GB, fits. The data is the reason. With 900 records, a method that can change all 134,515,008 weights has more room to overfit and to forget. The adapter keeps the base model safe and removable. And no method learns a rule from examples that do not exist: zero records show rules H2 to H5 or a French ticket. More trainable numbers do not add examples.

What would change the answer: suppose that the training loss and the validation loss both stay high. Then the adapter is too small to fit even the training data. Raise the rank, or add layers, and measure again. Module 4 trains rank 8 and shows its curves.

Your turn: an adapter on a feed-forward layer

Kwame asks what rank 8 on gate_proj alone would train. Predict it by hand from the shape in the output above. Then run:

python -m adapt params --rank 8 --targets gate_proj

Show the answer

gate_proj is 1536 × 576. One block trains 8 × (576 + 1536) = 16,896 numbers, and 30 blocks train 506,880. The command prints:

LoRA rank 8 on gate_proj                        506,880    0.38%        2.0

One wide feed-forward layer trains more than q_proj and v_proj together (460,800), because its out side is 1,536.

Common mistake

A layer name that does not exist

Symptom: Sam wants the query and value layers and types short names:

python -m adapt params --rank 8 --targets q v

LoRA rank 8 on q, v                                   0    0.00%        0.0

There is no error. The adapter trains nothing.

Cause: the names must match the model’s layer names exactly. No layer is called q or v, so no layer gets an adapter.

Fix: copy the names from the line Linear layers in one block: --targets q_proj v_proj. Before a training run, check that the trainable count is not zero. Module 4’s train prints it first: 460,800 of 134,975,808 parameters trainable (0.34%). The total includes the adapter.

Knowledge check

Check your understanding

Which numbers change in full tuning, and which in LoRA? For SmolLM2-135M, how large is the file that each method saves for one version?

Show the answer

Full tuning may change all 134,515,008 weights, so each version is a whole new model: 538.1 MB in float32. LoRA freezes every weight of the model. It adds two thin matrices, A and B, next to the chosen layers, and trains only them. Rank 8 on q_proj and v_proj trains 460,800 numbers: a file of 1.8 MB, used together with the unchanged base model.

Check your understanding

Predict: the course changes rank 8 to rank 32, on the same two layers. How many numbers does the adapter train, and how large is its file? What happens to the base model?

Show the answer

The count follows the rank: 32 is 4 times 8, so 4 × 460,800 = 1,843,200 numbers. In float32 that is 1,843,200 × 4 bytes, about 7.4 MB. You can check it with python -m adapt params --rank 32. The base model does not change: its 134,515,008 weights stay frozen at any rank.

Check your understanding

Tomás wants rank 64 because 'more trainable numbers learn more'. Which evidence would make you raise the rank? Which evidence is not a reason to raise it?

Show the answer

A reason: the training loss and the validation loss both stay high. Then the adapter cannot fit even its training examples, and more capacity may help, so measure a higher rank. Not a reason: mistakes on cases that no training record shows, such as French tickets or rules H2 to H5. A larger adapter has no examples to learn them from. Those need new data, or another method, not a higher rank.

Recap

• Full fine-tuning may change every weight: 134,515,008 numbers and a 538.1 MB copy per version for SmolLM2-135M. With few examples, it risks catastrophic forgetting.
• An adapter freezes the model and trains a few new numbers. LoRA builds each change from two thin matrices, A (rank × in) and B (out × rank): rank × (in + out) numbers instead of out × in.
• The course’s adapter, rank 8 on q_proj and v_proj, trains 460,800 numbers (0.34%): a 1.8 MB file. python -m adapt params counts it from model/config.json.
• The data, not the memory, decided the method. No rank adds examples that the data does not have.

Next, you estimate the memory that training needs, and see what quantization and other methods change.

Next: Memory, quantization and other methods