CLIP zero-shot
- Lesson 10
- 35 min
- Module 3: Embeddings, CLIP and few labels
Where we are
In the last lesson, embeddings put similar photos close together, but every method still needed labelled training photos to compare with. CLIP goes one step further: it puts photos and text in the same space. In this lesson, we classify catalogue photos with text prompts and no training, and we see how much the wording of a prompt matters. First, a worker with only a written list.
Lesson 10 of 30 · Vision AI: See, Count and InspectCLIP zero-shot
1. Seeing is a business problem
2. Baselines and pretrained classifiers
3. Embeddings, CLIP and few labels
4. Detect and count
6. Inspect for defects
7. Match, search and describe
8. Run vision in production
What you will be able to do
Classify photos into Larkfield’s categories with only the names of the categories, and no labelled photo. This is how you can sort photos on the first day of a new category, before anyone has labelled anything.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. Its setup cells install the packages, with PyTorch and open_clip, and download the catalogue photos as zip files. It downloads the CLIP model (605 MB) once. It needs no account and no key. About 30 minutes, plus a few minutes for the first download.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Embeddings, the cosine similarity and the confused pairs plant pot–vase and bowl–mug, from Image embeddings.
- Accuracy and recall per class, from Classification metrics in the machine learning course.
- The validation set and the test set, from Train, validation and test sets.
What you need on your computer
Nothing for the notebook route. The CLIP download needs about 1 GB of free disk space. Your larkvision project does not change in this lesson.
Tested with Python 3.12.13, PyTorch 2.15.0 and open_clip 4.2.1 on macOS, in Jupyter. Colab, Kaggle and Windows were not tested.
Sorting from a written list
Section titled “Sorting from a written list”On Monday, Larkfield’s returns warehouse gets a new shelf for a product type it never had before. Hamid has no time to show the new worker any examples. He gives her a list instead: “Shelf 7: plant pots. Shelf 8: vases.” She has never seen Larkfield’s plant pots. She still knows what the words mean, because she has seen plant pots and read about them all her life. So she reads the word, looks at the item, and decides.
This works because she connects two kinds of knowledge: what things look like and what they are called. A child who has only seen photos of a giraffe in a book can point at one in the zoo. Nobody needed to show her a hundred labelled giraffes first.
The problem: a category with no labelled photos
Section titled “The problem: a category with no labelled photos”The embeddings of the last lesson are good descriptions, but every classifier we built with them needed labelled training photos. The nearest-photo rule needs photos to compare with. The last layer of transfer learning needs a few hundred labels.
Omar, the product manager, often adds a new category. On its first day, it has zero labelled photos. Labelling 50 photos per class takes a person hours, and the photos must first be collected. Until then, the methods of this course so far have nothing to learn from. Can we sort photos into a category that has only a name?
The idea: one space for photos and words
Section titled “The idea: one space for photos and words”CLIP has two encoders. The image encoder turns a photo into 512 numbers, as in the last lesson. The text encoder turns a sentence into 512 numbers too, in the same space. A photo of a plant pot and the sentence “a photo of a plant pot” get embeddings that point in similar directions.
CLIP learned this from photos with their captions from the web. The model that this course uses was trained by LAION on about two billion such pairs (its model card says so). In training, CLIP sees a batch of photos and their captions. It learns to give each photo a high cosine with its own caption and a low cosine with all the other captions in the batch. This is contrastive training: it learns by contrast between right and wrong pairs. The name says it: CLIP is short for Contrastive Language–Image Pre-training.
To classify a photo, we write one sentence per class, called a prompt. For example: “a photo of a bowl.”, “a photo of a candle.”. We embed the eight prompts and the photo. The class whose prompt has the highest cosine with the photo is the answer. No labelled photo was used, so this is called zero-shot classification: zero examples.
Text description of the diagram
On the left, a photo of a plant pot goes into the image encoder, which gives 512 numbers. Below it, eight texts go into the text encoder, from “a photo of a bowl.” to “a photo of a vase.”. It gives 512 numbers for each text. Both arrows meet in one shared space. There, we compute the cosine of the photo with each text. It is 0.269 for plant pot, 0.211 for vase, 0.210 for bowl, and lower for the others. On the right, the answer is the text with the highest cosine: plant pot.
A tiny worked example: two plant pots
Section titled “A tiny worked example: two plant pots”Here are two test photos. Both are labelled plant pot.

Text description of the diagram
Photo 1 shows an aloe plant with thick green leaves growing in dark soil, seen from above. Photo 2 shows a big bunch of sunflowers in a white plastic bucket, the bucket you met among the neighbours in the last lesson.
Section 2 of the notebook computes the cosine of each photo with the eight prompts “a photo of a ….”. The four highest:
| Photo 1 | Cosine | Probability | Photo 2 | Cosine | Probability | |
|---|---|---|---|---|---|---|
| plant pot | 0.269 | 0.994 | vase | 0.244 | 0.563 | |
| vase | 0.211 | 0.003 | plant pot | 0.241 | 0.434 | |
| bowl | 0.210 | 0.003 | mug | 0.187 | 0.002 | |
| candle | 0.193 | 0.000 | bowl | 0.173 | 0.000 |
The cosines between a photo and a text are small, about 0.1 to 0.3. Photos and sentences are different things, so they never point in exactly the same direction. What matters is which prompt is highest. For photo 1, plant pot wins clearly. For photo 2, vase wins by 0.003.
To turn cosines into probabilities, CLIP multiplies them by 100 and applies the softmax. The softmax raises e to each number, then divides each result by their sum. Take photo 2’s two highest values, which hold almost all the probability:
- 100 × 0.244 = 24.4 and 100 × 0.241 = 24.1. The difference is 0.3.
- e to the power 0.3 is 1.35. So vase gets 1.35 times the probability of plant pot.
- 1.35 / (1.35 + 1) = 0.57 for vase, and 1 / 2.35 = 0.43 for plant pot. The exact values above (0.563 and 0.434) also give a little to mug.
The factor 100 makes small differences in cosine into large differences in probability. So a probability of 0.56 here does not mean “56% sure” in an everyday sense. It means “a close call between two classes”. Photo 2 is a close call for a good reason: cut sunflowers in a bucket are used like a vase. A person might hesitate too.
The words of the prompt matter
Section titled “The words of the prompt matter”The prompt is all that CLIP knows about Larkfield’s classes. So its words matter. Section 3 of the notebook compares three ways to write the prompts on the 160 test photos:
- Class name only: “plant pot”.
- “a photo of a {class}.”: one sentence, like the captions that CLIP learned from.
- Eight templates: eight sentences per class, such as “a product photo of a plant pot.” and “a close-up photo of a plant pot.”. CLIP embeds all eight, and the mean of the eight embeddings is the class’s text embedding. This is a prompt set, also called prompt ensembling.
- Accuracy, 160 test photos
- Plant pot recall, 20 test photos
Text description of the chart
A column chart with two groups of three columns, on an axis from 0.7 to 1. Accuracy on the 160 test photos: class name only 0.900, “a photo of a” 0.906, eight templates 0.925. Plant pot recall on its 20 test photos: class name only 0.90, “a photo of a” 0.80, eight templates 0.80.
Many guides say that “a photo of a plant pot” works better than “plant pot”, because CLIP’s training captions are sentences, not single words. Here, that advice did not hold. The overall accuracy rose by one photo (0.900 to 0.906). Plant pot got worse: 18 of 20 right with the name alone, 16 of 20 with “a photo of a”. The eight templates did best overall, with 0.925: four more photos right than the name alone. They did not help plant pot either.
Be careful with these differences. The test set has 160 photos, so one photo moves the accuracy by 0.006. A difference under about 3 points can be chance. The eight templates win by 2.5 points: a small, real gain at best. The lesson is not “this wording is best”. The lesson is: measure the wording on your own photos, and prefer a prompt set to one sentence.
What zero-shot gets wrong
Section titled “What zero-shot gets wrong”With “a photo of a {class}.”, these are the mistakes among the pot classes:

Text description of the diagram
Five photos. Sunflowers in a white bucket, labelled plant pot: CLIP said vase. A wire basket with tulips and painted eggs, labelled plant pot: CLIP said vase. A round glass bowl of succulents seen from above, labelled plant pot: CLIP said bowl. A painted ceramic pot with a plant, on a dark table, labelled plant pot: CLIP said bowl. A white, cracked jug with a pink flower, among plants, labelled vase: CLIP said plant pot.
Look at them as Hamid’s worker would. Each mistake is a container with two honest names: a glass bowl full of plants, a bucket of flowers, a jug among plants. CLIP’s words come from the web, where people use “vase”, “bowl” and “pot” loosely. Larkfield’s catalogue team uses them in its own way. Zero-shot cannot learn that way: only labelled examples can teach it. That is the job of the next lesson.
Larkfield: zero labels against 453 labels
Section titled “Larkfield: zero labels against 453 labels”Put zero-shot next to the methods of Module 2 in the comparison table. All rows are on the same 160 test photos:
| Method | Labelled photos | Accuracy | Worst class |
|---|---|---|---|
| Colour histogram + 5-NN | 453 | 0.263 | vase 0.05 |
| ResNet-18, frozen + last layer | 453 | 0.812 | candle 0.45 |
| ResNet-18, fine-tuned 4 epochs | 453 | 0.850 | candle 0.40 |
| ConvNeXt-Atto, frozen + last layer | 453 | 0.887 | candle 0.80 |
| ResNet-50, frozen + last layer | 453 | 0.894 | plant pot 0.80 |
| CLIP zero-shot, eight templates | 0 | 0.925 | plant pot 0.80 |
CLIP, with no label at all, is at least as good as every Module 2 method that used all 453 labels. Its lead over ResNet-50 is 5 photos, about 3 points: at the edge of noise. It also does not suffer from the small candle class: it found all 20 test candles, because it never learned from the unbalanced training photos. On the warmer new-studio photos, it gives 0.912, almost the same.
Tomás is delighted: “So we never need to label anything again.” Priya is careful: “Zero-shot knows the web’s meaning of our words. When our meaning is different, as for plant pots, it stays wrong, and we have no way to teach it. And 0.925 means 12 wrong photos in 160.”
When to use zero-shot, and what it costs
Section titled “When to use zero-shot, and what it costs”Use it:
- On the first day of a new category, before any photo is labelled.
- To sort a large pile of photos roughly, so that people label faster: they correct CLIP’s answer instead of starting from nothing.
- As a baseline that any trained method must beat.
Do not use it alone:
- When your classes follow your own rules, not everyday words. Plant pot against vase is Larkfield’s rule.
- For small details. The comparison table has a CLIP prompt for “a damaged tea light candle”. It costs about 1,013 dollars a day on the returns test. That is better than no inspection (2,229), but far from the best method of Module 6. A whole-photo embedding hardly notices a small chip.
- For words that CLIP never saw in captions, such as Larkfield’s product codes.
Costs: the model is 605 MB and needs about 8 ms per photo on a laptop CPU, in batches. The text embeddings are computed once per class. The weights are under the MIT licence. They learned from web captions, with the web’s mistakes and biases. Module 7 returns to the licences and the risks of such models.
Guided practice: zero-shot in the notebook
Section titled “Guided practice: zero-shot in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The first installs the packages, with open_clip. The second ends with a
Ready:line. - Run section 1. It downloads CLIP once. You see
(120, 512) (160, 512). - Run the next cell. The live embedding of photo 1 and the stored one have a cosine of
1.0: they are the same model. - In section 2, predict the two closest classes for photo 2. Then run the cell. You see vase
0.563and plant pot0.434. - Run section 3. You see the accuracies
0.9,0.906and0.925, and plant pot recall0.9,0.8and0.8.
Your turn: another name for one class
Section titled “Your turn: another name for one class”Open Images, where the catalogue photos come from, calls this class “Flowerpot”. Maybe CLIP knows that name better. In section 4 of the notebook, replace “plant pot” with “flowerpot” in the class names. Then measure the accuracy with the eight templates on the validation photos, and put it in flowerpot_val. Before you run it, predict: will “flowerpot” win?
Show the answer
names = NAMES.copy()names[CLASSES.index("plant_pot")] = "flowerpot"flowerpot_val = round(float(np.mean(np.array(zero_shot(Xval, names, EIGHT)) == val["label"].values)), 3)print(flowerpot_val)0.967The next cell measures “plant pot” on the same validation photos: also 0.967. Both names get 116 of the 120 validation photos right. The validation set cannot choose between them. So you keep the simpler choice, the name your team uses, and you do not look at the test set to break the tie. The failure case below shows why.
Common mistake
Choosing the prompt on the test set
Symptom: Tomás tries five names for the plant pot class with the eight templates, and scores each on the 160 test photos:
| Name for plant pot | Test accuracy | Validation accuracy |
|---|---|---|
| plant pot | 0.925 | 0.967 |
| flowerpot | 0.938 | 0.967 |
| flower pot | 0.938 | 0.950 |
| pot plant | 0.944 | 0.967 |
| houseplant in a pot | 0.931 | 0.958 |
He keeps “pot plant” and reports 0.944 to Omar: “Two points better than the course’s prompts.”
Cause: he chose the name because of its test score. Some of that gain is luck: these 160 photos happen to suit “pot plant”. The best of five tries is always a little too good, as Train, validation and test sets showed with 100 random rules. The test score is no longer an honest estimate for new photos. On the validation photos, the five names are within one or two photos of each other: there, no name is clearly better.
Fix: compare prompts on the validation photos only. When validation cannot tell them apart, keep the simplest prompt, and do not break the tie with the test set. Score the final choice on the test set once. To find the best name, label more validation photos first. 120 photos cannot show a difference of one or two photos.
Consequence in AI work: prompt wording feels free to change, so teams change it often, and they check each change on the same small set. After twenty changes, the score on that set says more about the set than about the prompt. A prompt is a setting of the model, like any other: choose it on validation, report it on test.
Knowledge check
Section titled “Knowledge check”Check your understanding
Explain in your own words how CLIP classifies a photo into one of eight classes with no labelled photo. Why is this called zero-shot?
Show the answer
CLIP has an image encoder and a text encoder that put photos and sentences in the same space. It learned this from about two billion photos with captions. In training, each photo gets a high cosine with its own caption and a low cosine with the others. To classify, you write one prompt per class, such as “a photo of a vase.”. You embed the eight prompts and the photo, and take the class whose prompt has the highest cosine. It is zero-shot because the classifier saw zero labelled examples of Larkfield’s classes: only their names.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Photo 2 (sunflowers in a bucket) gets vase 0.563 and plant pot 0.434. Tomás says: 'CLIP is 56% sure that it is a vase.' What would you tell him?
Show the answer
The two cosines are 0.244 and 0.241: almost equal. CLIP multiplies them by 100 before the softmax, so a difference of 0.003 in cosine becomes 0.56 against 0.43. The numbers mean “a close call between vase and plant pot”, not a measured chance of 56% of being right. They are not calibrated probabilities. The photo is also honestly ambiguous: cut flowers in a bucket are used like a vase. Such a photo should go to a person, or to the labelling guide.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Omar adds a new category, garden lanterns, tomorrow. No photo is labelled yet. What would you do on day 1 with CLIP, and why would you not stop there?
Show the answer
On day 1, add a prompt set for “garden lantern” (several templates), and use CLIP zero-shot to sort the new photos roughly. People can then correct its answers, which is faster than labelling from nothing. Measure it on a small validation set as soon as one exists. Do not stop there: zero-shot follows the web’s meaning of the words. If Larkfield’s lanterns overlap with its lamps or candles in Larkfield’s own way, CLIP cannot learn that rule. Also, its accuracy here (0.925) still means 12 wrong photos in 160. With a few labelled photos per class, a small trained classifier can learn Larkfield’s line: the next lesson.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- CLIP embeds photos and sentences in one space. It learned this with contrastive training on about two billion photo–caption pairs.
- Zero-shot: write one prompt per class, and take the prompt with the highest cosine. No labelled photo is needed.
- CLIP’s cosines are small (about 0.1 to 0.3). The factor 100 and the softmax turn small differences into large probabilities: read them as “clear” or “close call”, not as calibrated chances.
- On Larkfield’s test photos: class name only 0.900, “a photo of a” 0.906, eight templates 0.925. “A photo of a plant pot” did not beat “plant pot” here (plant pot recall 0.90 → 0.80). Measure the wording on your own photos.
- With zero labels, CLIP is at least as good as every Module 2 method that used 453 labels (best 0.894). Its mistakes are containers that Larkfield names in its own way.
- Choose prompts on the validation photos, never on the test set.
You can now classify with no labels at all, and with all of them. But zero-shot cannot learn Larkfield’s own rules, and full training needs many labels. Next, you try the cheap middle way: a small classifier on embeddings, with 5, 20 or all photos per class.