Linear probes and few-shot
- Lesson 11
- 40 min
- Module 3: Embeddings, CLIP and few labels
Where we are
CLIP classified photos with no labels at all, but it cannot learn Larkfield's own line between a plant pot and a vase. Transfer learning used all our labels and a training run. Between the two, there is a cheap middle way. In this lesson, we train a logistic regression on embeddings with 1, 5, 20 and all images per class. First, a few example photos pinned above each shelf.
Lesson 11 of 30 · Vision AI: See, Count and InspectLinear probes and few-shot
1. Seeing is a business problem
2. Baselines and pretrained classifiers
3. Embeddings, CLIP and few labels
4. Detect and count
6. Inspect for defects
7. Match, search and describe
8. Run vision in production
What you will be able to do
Teach a classifier Larkfield’s own categories from a few labelled photos per class, in about a second of training. Find out how many labels are enough, and fix a class that has too few photos.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. Its setup cells install NumPy, pandas, scikit-learn and matplotlib, and download the catalogue photos with their stored embeddings as zip files. It downloads no model. It needs no account and no key. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Embeddings and the three models CLIP, DINOv2 and ResNet-18, from Image embeddings.
- CLIP zero-shot and its 0.925 with no labels, from CLIP zero-shot.
- A frozen ResNet-18 with a new last layer (0.812) and fine-tuning (0.850), from Transfer learning.
- What a logistic regression does, from Classification intuition in the machine learning course. This lesson recalls the main idea.
What you need on your computer (only for the project route; the notebook needs none of it)
- Your
larkvisionfolder besidenextia-learning-labs, at the snapshotend-of-m02:python -m pytest -qprints11 passed. This lesson adds two files to it. - In each new terminal: go to
larkvision, activate.venv, and setLARKVISION_DATAagain, as in Meet the Larkfield images.
Tested with Python 3.12.13 and scikit-learn 1.9.1 on macOS, in Jupyter and in a terminal. The Windows commands, Colab and Kaggle were not tested.
A few examples above each shelf
Section titled “A few examples above each shelf”Hamid finds a cheap way to train new workers. Above each shelf, he pins five photos of products that belong there. One photo above the plant pot shelf shows a pot with soil and a small plant. One above the vase shelf shows a tall glass with cut flowers. The new worker already knows how to look at things: shapes, materials, what is inside. The five photos only show her where Larkfield draws the line between two shelves.
She does not need a long course on how to see. She needs a few examples of each shelf, because seeing is the hard part, and she already learned it.
The problem: between no labels and many labels
Section titled “The problem: between no labels and many labels”The last two lessons gave us two extremes. CLIP zero-shot needs no label, and reached 0.925. But it cannot learn Larkfield’s rules: its mistakes were containers that the catalogue team names in its own way, and no prompt fixes that. Transfer learning in Module 2 learned from all 453 labels. Fine-tuning ResNet-18 took about half a minute of training and reached only 0.850.
We want the worker’s way. Keep a model that already knows how to see, and learn only Larkfield’s lines from a few labelled photos.
The idea: a linear probe
Section titled “The idea: a linear probe”The embedding is the worker’s way of seeing. It is already computed and stored for every photo. On top of it, we train the simplest classifier that can learn a line: a logistic regression. For each class, it learns one weight for each of the 512 numbers, and one extra number, the bias. To classify a photo, it multiplies the embedding by each class’s weights, adds the bias, and gets one score per class. The softmax turns the eight scores into probabilities, as in the last lesson.
A logistic regression on frozen embeddings is called a linear probe. “Linear” because each score is a weighted sum: it can only draw straight boundaries in the space. “Probe” because it tests what the embedding already knows, as a doctor’s probe tests what is under the skin. If a straight boundary separates the classes well, the embedding already holds the information.
Text description of the diagram
The diagram has four boxes in a row (one under the other on a phone). The first box holds the labelled photos: a few per class, for example 5, 20 or all. The second box, grey and dashed, is the frozen model. CLIP, DINOv2 or ResNet-18 turns each photo into an embedding, and does not change. The third box is the linear probe: a logistic regression. It computes 8 scores, each a sum of weights times the 512 numbers, plus a bias. It trains in about one second. The last box is the answer: the class with the highest probability, from the softmax of the 8 scores.
Compare it with the frozen ResNet-18 of Module 2. That was a linear probe too: a new last layer on frozen features. The difference here is the embedding: CLIP and DINOv2 learned from far more photos, and with other training methods.
A tiny worked example
Section titled “A tiny worked example”Here is a probe with only two classes, mug and vase, and embeddings of only two numbers. The numbers are made up for this example. Training gave these weights and biases:
| Class | Weight 1 | Weight 2 | Bias |
|---|---|---|---|
| mug | 3 | −2 | 0 |
| vase | −1 | 4 | 0 |
A new photo has the embedding (0.8, 0.6). The score of each class is weight 1 × number 1 + weight 2 × number 2 + bias:
- mug: 3 × 0.8 + (−2) × 0.6 + 0 = 2.4 − 1.2 = 1.2
- vase: (−1) × 0.8 + 4 × 0.6 + 0 = −0.8 + 2.4 = 1.6
The softmax: e1.2 = 3.32 and e1.6 = 4.95, with a sum of 8.27. So vase gets 4.95 / 8.27 = 0.60, and mug gets 0.40. The probe says vase.
Training is the search for the weights and biases that give the highest score to the right class on the labelled photos. For Larkfield, the probe learns 8 × 512 weights + 8 biases = 4,104 numbers. scikit-learn finds them in about one second on a laptop. Fine-tuning the last block of ResNet-18, as in Module 2, changes about 8.4 million numbers.
Larkfield: a probe on all the labels
Section titled “Larkfield: a probe on all the labels”Priya trains a CLIP probe on all 453 training photos. You can do it in section 2 of the notebook, or with the project, below. The result on the 160 test photos:
CLIP ViT-B/32 (OpenCLIP, laion2b_s34b_b79k) + logistic regression (all labels, 453 labelled photos): accuracy 0.894; worst class candle (recall 0.60)The accuracy, 0.894, is as good as the best Module 2 model (ResNet-50, 0.894), but lower than CLIP zero-shot (0.925). The reason is in the worst class: candle. The probe finds only 12 of the 20 test candles. Yet when it says “candle”, it is always right: its precision is 1.0.
Candle has only 15 training photos, against 41 to 70 for the other classes: a problem we planted. A logistic regression minimises its mistakes on the training photos. Candles are 15 of 453 photos, about 3%. Getting them wrong costs the probe little. So it learns that “candle” is a rare answer, and says it only when it is very sure.
Balanced classes
Section titled “Balanced classes”The fix is to tell the probe that each class matters the same. With balanced class weights, each class gets the same total weight in training. A candle photo then counts as much as about 4.7 chair photos, because the chair class has 70 photos and the candle class 15. In scikit-learn, it is one setting: class_weight="balanced".
CLIP ViT-B/32 (OpenCLIP, laion2b_s34b_b79k) + logistic regression (all labels, balanced classes, 453 labelled photos): accuracy 0.944; worst class bowl (recall 0.85)The accuracy rises from 0.894 to 0.944: 8 more test photos right. Candle recall jumps from 0.60 to 0.95. The chart shows the classes that changed:
- All labels
- All labels, balanced classes
Text description of the chart
A column chart with three pairs of columns, on an axis from 0.5 to 1. Candle recall: 0.60 with all labels, 0.95 with balanced classes. Cushion recall: 0.95 and 1.00. Accuracy: 0.89 and 0.94. The six other classes have the same recall in both: bowl 0.85, chair 1.00, lamp 0.95, mug 0.90, plant pot 0.90 and vase 1.00.
Only candle changes much. Cushion gains one photo, and the six other classes do not move. The same fix helps the other embeddings less. DINOv2’s candle recall goes from 0.70 to 0.80. ResNet-18’s goes from 0.45 to 0.70. A better embedding gives the probe more to work with.
How many labels does a probe need?
Section titled “How many labels does a probe need?”Omar asks the question that decides the budget: “How many photos must we label for each category?” Priya answers with a learning curve. She trains the probe with 1, 5, 10, 20, 40 and all labelled photos per class, chosen at random. Each size is drawn 10 times, and the chart shows the mean accuracy. Learning from a few examples per class is called few-shot learning.
- CLIP probe
- DINOv2 probe
- ResNet-18 probe
- CLIP zero-shot (no labels)
- ResNet-18 fine-tuned (all labels)
Text description of the chart
A line chart. The x axis has six steps: 1, 5, 10, 20, 40 and all labelled photos per class. The y axis is the test accuracy, from 0.5 to 1. The CLIP probe: 0.624 with 1 photo per class, 0.869 with 5, 0.904 with 10, 0.929 with 20, 0.912 with 40 and 0.894 with all. The DINOv2 probe: 0.660, 0.850, 0.884, 0.893, 0.887 and 0.887. The ResNet-18 probe: 0.637, 0.827, 0.857, 0.848, 0.832 and 0.806. A dashed horizontal line at 0.925 is CLIP zero-shot with no labels. A single dot at “all”, at 0.850, is the ResNet-18 fine-tuned for 4 epochs in Module 2.
Read the chart from left to right:
- One photo per class is not enough: 0.624 for CLIP, and from 0.487 to 0.762 depending on which photo was drawn. One example cannot show where a class ends.
- Five photos per class give 0.869 with CLIP: already better than fine-tuning ResNet-18 on all 453 photos (0.850).
- Twenty photos per class give 0.929: as good as zero-shot (0.925), and now the probe can learn Larkfield’s own rules. That is 155 labelled photos in all, because candle has only 15.
- More labels make it worse. With 40 per class, 0.912; with all, 0.894. The classes become unbalanced again: the others grow to 41–70 photos, and candle stays at 15.
This last point surprises most people. It is the same cause as before. The probe cares about the balance between classes, not only about the number of photos. With balanced class weights, all the labels give 0.944, the best result of this module.
The hidden test, once
Section titled “The hidden test, once”The course keeps a hidden catalogue test of 160 more photos, as Meet the Larkfield images explained. Priya chose between the probe with all labels and the probe with balanced classes on the validation photos: 0.925 against 0.933. She kept the balanced probe, and the course team scored it once on the hidden test: 0.956, with candle as the worst class at 0.90.
The hidden score is a little higher than the test score (0.944). With 160 photos, that difference is about two photos: chance, not a better model. The important result is that the choice made on validation held on photos that nobody had seen.
Add the probe to your project
Section titled “Add the probe to your project”The project gets the probe as a command, vision probe. You copy two new files from the snapshot end-of-m03. Check the place first: you must be in your larkvision folder, with the virtual environment active.
-
Open a terminal and go to your
larkvisionfolder. Activate the virtual environment:source .venv/bin/activateon macOS and Linux,.venv\Scripts\Activate.ps1on Windows. -
If this is a new terminal, set
LARKVISION_DATAagain. It lives only in the terminal where you set it. The command is step 5 of The project on your computer. -
Copy the two files of this lesson from the snapshot:
Terminal window S=../nextia-learning-labs/C42/snapshots/end-of-m03for f in larkvision/embeddings.py larkvision/commands/probe.py; do cp "$S/$f" "$f"; doneTerminal window $S = "..\nextia-learning-labs\C42\snapshots\end-of-m03"foreach ($f in 'larkvision\embeddings.py', 'larkvision\commands\probe.py') { Copy-Item "$S\$f" $f } -
Run the tests. You still see
11 passed: the tests of the new commands come with the next lesson.Terminal window python -m pytest -q -
Train the CLIP probe on all the labels. It takes about one second:
Terminal window vision probe --embeddings clip -
Train it again with balanced classes:
Terminal window vision probe --embeddings clip --balanced -
Train it with 20 photos per class:
Terminal window vision probe --embeddings clip --per-class 20
Steps 5 and 6 print the two lines that you saw above. Step 7 prints:
CLIP ViT-B/32 (OpenCLIP, laion2b_s34b_b79k) + logistic regression (20 per class, 155 labelled photos): accuracy 0.938; worst class bowl (recall 0.90)This is one random draw (--seed 0), so it gives 0.938, not the mean of 10 draws (0.929). With --seed 1, the same command gives 0.925. larkvision/embeddings.py loads the stored embeddings by the model’s name and scales them to length 1. larkvision/commands/probe.py holds the function probe, with the same settings as the notebook. Each run adds a row to your own table: vision compare --task catalogue shows it at the end, marked “(yours)”.
When to use a probe, and what it costs
Section titled “When to use a probe, and what it costs”Use a linear probe when a good embedding exists for your photos and you have, or can get, about 20 labelled photos per class. It is the first trained method to try. It trains in a second, needs no GPU, and is easy to train again when a label changes.
Its limits:
- It cannot see more than the embedding. If the embedding mixes two classes, a straight line cannot separate them. Fine-tuning changes the embedding itself, and can.
- It follows the balance of its labels. A class with few photos gets few answers. Use balanced class weights, or label the small class first.
- Balanced weights have a price. They make the probe say the small class more often. Here, nothing else got worse. With a very small class, they can add false candles.
- It needs the embedding model at prediction time. A new photo must go through CLIP (605 MB) before the probe. The probe itself is a few kilobytes.
Guided practice: the probe in the notebook
Section titled “Guided practice: the probe in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The second one ends with a
Ready:line. - Run section 1. You see
453 training photos; 15 of them are candles. - In section 2, predict whether candle is the best or the worst class. Then run the cell. You see accuracy
0.894and candle recall0.60. - Run section 3. You see accuracy
0.944and candle recall0.95. - In section 4, predict whether the accuracy rises all the way to “all labels”. Then run the cell. It takes about 30 seconds and draws the learning curve.
The notebook prints 0.928 for 20 photos per class, and the chart above says 0.929. The course’s stored embeddings are rounded to save space, and one draw of ten moves by one photo. Treat the two as the same.
Your turn: DINOv2 with 20 photos per class
Section titled “Your turn: DINOv2 with 20 photos per class”Does DINOv2, which learned from photos alone, do as well as CLIP with 20 photos per class? Predict first. In section 5 of the notebook, use the function curve with "dinov2" and the size 20 only, and put the mean accuracy in dinov2_20. On the project route, run vision probe --embeddings dinov2 --per-class 20 for one draw: it gives 0.900.
Show the answer
dinov2_20 = curve("dinov2", sizes=(20,))[20]print(dinov2_20)0.893DINOv2 with 20 photos per class reaches 0.893, against 0.929 for CLIP. On these product photos, CLIP’s embedding separates the classes better. That is not always so. To find the same product in a customer’s photo, DINOv2 is far better than CLIP: see Find the same product. Which embedding is best depends on the task, so measure it on yours.
Common mistake
One lucky draw
Symptom: Tomás labels 5 photos per class, trains a probe once and gets this:
CLIP ViT-B/32 (OpenCLIP, laion2b_s34b_b79k) + logistic regression (5 per class, 40 labelled photos): accuracy 0.819; worst class vase (recall 0.45)He tells Omar: “Five photos per class are useless: vase recall is 0.45.” Priya is not convinced. She knows that another 5 photos per class could give a very different result.
Cause: with 5 photos per class, the result depends on which 5 photos you label. One draw of a plant pot that looks like a vase can move a whole class. Over 10 random draws, the CLIP probe with 5 photos per class went from 0.819 to 0.925, with a mean of 0.869. Tomás’s draw was the worst of the ten.
Fix: for a few-shot result, train on several random draws and report the mean and the range, as the learning curve does. When you choose which photos to label, choose varied ones: different shapes, materials and backgrounds. Do not choose the five easiest.
Consequence in AI work: a decision about a labelling budget made on one draw can be wrong in both directions. Omar could cancel a method that works, or trust one that only worked by luck.
Knowledge check
Section titled “Knowledge check”Check your understanding
What is a linear probe, and what does it learn? Why does it train in a second while fine-tuning takes minutes?
Show the answer
A linear probe is a logistic regression trained on frozen embeddings. It learns one weight per embedding number and one bias for each class: here 8 × 512 + 8 = 4,104 numbers. The model that makes the embeddings does not change, and the embeddings are computed once and stored. Fine-tuning changes the network itself, about 8.4 million numbers for the last block of ResNet-18, and must run every photo through it many times. So the probe trains in about one second, and fine-tuning in about half a minute on these 453 photos, and much longer on more.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
With all 453 labels, the CLIP probe finds only 12 of the 20 test candles, and its precision for candle is 1.0. Explain both numbers, and say what fixes the recall.
Show the answer
Candle has only 15 of the 453 training photos, about 3%. The logistic regression minimises its mistakes on the training photos, where candles count little, so it learns that candle is a rare answer. It says candle only when it is very sure. So it is always right when it says it (precision 1.0), but it misses 8 candles (recall 0.60). Balanced class weights give each class the same total weight, so a candle photo counts as much as about 4.7 chair photos. With them, candle recall rises to 0.95 and accuracy to 0.944.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Omar plans a new category. Using the learning curve, how many photos per class would you ask him to label first, and why not 'as many as possible'?
Show the answer
Ask for about 20 varied photos for the new category first. With 20 per class, the CLIP probe reached 0.929 (mean of 10 draws), as good as zero-shot. And it can learn Larkfield’s own rules. With 5 per class it reached 0.869, and one draw gave as little as 0.819. “As many as possible” in one class is not better: the curve fell to 0.894 with all labels, because the classes became unbalanced. Keep the classes balanced, or use balanced class weights. Then measure the new category’s recall on validation photos before you add more labels.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A linear probe is a logistic regression on frozen embeddings: 4,104 numbers to learn, about one second of training.
- On all labels, the CLIP probe reaches 0.894, with candle recall 0.60: the class with 15 photos gets few answers.
- Balanced class weights fix it: 0.944, candle recall 0.95. Chosen on validation, the balanced probe scored 0.956 on the hidden test, once.
- Few-shot: 5 photos per class give 0.869, 20 give 0.929, all give 0.894. More labels lowered the accuracy because the classes became unbalanced.
- A few-shot result depends on the draw: report the mean and the range of several draws.
- Your project now has
vision probe, with--balancedand--per-class.
You can now decide how many labels a category needs. But the learning curve assumed that every label is right. Next, you find out how good the labels are. You meet two labellers who disagree, a labelling guide, and the label errors that we planted.