Skip to content
Français

CLIP zero-shot

  • Lesson 10
  • 35 min
  • Module 3: Embeddings, CLIP and few labels

Where we are

In the last lesson, embeddings put similar photos close together, but every method still needed labelled training photos to compare with. CLIP goes one step further: it puts photos and text in the same space. In this lesson, we classify catalogue photos with text prompts and no training, and we see how much the wording of a prompt matters. First, a worker with only a written list.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Classify photos into Larkfield’s categories with only the names of the categories, and no labelled photo. This is how you can sort photos on the first day of a new category, before anyone has labelled anything.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. Its setup cells install the packages, with PyTorch and open_clip, and download the catalogue photos as zip files. It downloads the CLIP model (605 MB) once. It needs no account and no key. About 30 minutes, plus a few minutes for the first download.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

Nothing for the notebook route. The CLIP download needs about 1 GB of free disk space. Your larkvision project does not change in this lesson.

Tested with Python 3.12.13, PyTorch 2.15.0 and open_clip 4.2.1 on macOS, in Jupyter. Colab, Kaggle and Windows were not tested.

On Monday, Larkfield’s returns warehouse gets a new shelf for a product type it never had before. Hamid has no time to show the new worker any examples. He gives her a list instead: “Shelf 7: plant pots. Shelf 8: vases.” She has never seen Larkfield’s plant pots. She still knows what the words mean, because she has seen plant pots and read about them all her life. So she reads the word, looks at the item, and decides.

This works because she connects two kinds of knowledge: what things look like and what they are called. A child who has only seen photos of a giraffe in a book can point at one in the zoo. Nobody needed to show her a hundred labelled giraffes first.

The problem: a category with no labelled photos

Section titled “The problem: a category with no labelled photos”

The embeddings of the last lesson are good descriptions, but every classifier we built with them needed labelled training photos. The nearest-photo rule needs photos to compare with. The last layer of transfer learning needs a few hundred labels.

Omar, the product manager, often adds a new category. On its first day, it has zero labelled photos. Labelling 50 photos per class takes a person hours, and the photos must first be collected. Until then, the methods of this course so far have nothing to learn from. Can we sort photos into a category that has only a name?

CLIP has two encoders. The image encoder turns a photo into 512 numbers, as in the last lesson. The text encoder turns a sentence into 512 numbers too, in the same space. A photo of a plant pot and the sentence “a photo of a plant pot” get embeddings that point in similar directions.

CLIP learned this from photos with their captions from the web. The model that this course uses was trained by LAION on about two billion such pairs (its model card says so). In training, CLIP sees a batch of photos and their captions. It learns to give each photo a high cosine with its own caption and a low cosine with all the other captions in the batch. This is contrastive training: it learns by contrast between right and wrong pairs. The name says it: CLIP is short for Contrastive Language–Image Pre-training.

To classify a photo, we write one sentence per class, called a prompt. For example: “a photo of a bowl.”, “a photo of a candle.”. We embed the eight prompts and the photo. The class whose prompt has the highest cosine with the photo is the answer. No labelled photo was used, so this is called zero-shot classification: zero examples.

A photo of a plant pot goes into the image encoder and becomes 512 numbers. Eight texts, a photo of a bowl to a photo of a vase, go into the text encoder and become 8 times 512 numbers. Both arrive in one shared space, where the cosine of the photo with each text is listed: plant pot 0.269, vase 0.211, bowl 0.210. The answer is the text with the highest cosine: plant pot.
CLIP zero-shot: the photo and the eight prompts are embedded in the same space, and the prompt with the highest cosine wins. The cosines are those of photo 1 below.
Text description of the diagram

On the left, a photo of a plant pot goes into the image encoder, which gives 512 numbers. Below it, eight texts go into the text encoder, from “a photo of a bowl.” to “a photo of a vase.”. It gives 512 numbers for each text. Both arrows meet in one shared space. There, we compute the cosine of the photo with each text. It is 0.269 for plant pot, 0.211 for vase, 0.210 for bowl, and lower for the others. On the right, the answer is the text with the highest cosine: plant pot.

Here are two test photos. Both are labelled plant pot.

Two photos. Photo 1: an aloe plant growing in dark soil, seen from above. Photo 2: a bunch of sunflowers in a white plastic bucket.
The two plant pots of the worked example, from the catalogue test photos. Photos: Ernie, mk97007, CC BY 2.0, from Open Images.
Text description of the diagram

Photo 1 shows an aloe plant with thick green leaves growing in dark soil, seen from above. Photo 2 shows a big bunch of sunflowers in a white plastic bucket, the bucket you met among the neighbours in the last lesson.

Section 2 of the notebook computes the cosine of each photo with the eight prompts “a photo of a ….”. The four highest:

Photo 1 Cosine Probability Photo 2 Cosine Probability
plant pot 0.269 0.994 vase 0.244 0.563
vase 0.211 0.003 plant pot 0.241 0.434
bowl 0.210 0.003 mug 0.187 0.002
candle 0.193 0.000 bowl 0.173 0.000

The cosines between a photo and a text are small, about 0.1 to 0.3. Photos and sentences are different things, so they never point in exactly the same direction. What matters is which prompt is highest. For photo 1, plant pot wins clearly. For photo 2, vase wins by 0.003.

To turn cosines into probabilities, CLIP multiplies them by 100 and applies the softmax. The softmax raises e to each number, then divides each result by their sum. Take photo 2’s two highest values, which hold almost all the probability:

  • 100 × 0.244 = 24.4 and 100 × 0.241 = 24.1. The difference is 0.3.
  • e to the power 0.3 is 1.35. So vase gets 1.35 times the probability of plant pot.
  • 1.35 / (1.35 + 1) = 0.57 for vase, and 1 / 2.35 = 0.43 for plant pot. The exact values above (0.563 and 0.434) also give a little to mug.

The factor 100 makes small differences in cosine into large differences in probability. So a probability of 0.56 here does not mean “56% sure” in an everyday sense. It means “a close call between two classes”. Photo 2 is a close call for a good reason: cut sunflowers in a bucket are used like a vase. A person might hesitate too.

The prompt is all that CLIP knows about Larkfield’s classes. So its words matter. Section 3 of the notebook compares three ways to write the prompts on the 160 test photos:

  • Class name only: “plant pot”.
  • “a photo of a {class}.”: one sentence, like the captions that CLIP learned from.
  • Eight templates: eight sentences per class, such as “a product photo of a plant pot.” and “a close-up photo of a plant pot.”. CLIP embeds all eight, and the mean of the eight embeddings is the class’s text embedding. This is a prompt set, also called prompt ensembling.
  • Accuracy, 160 test photos
  • Plant pot recall, 20 test photos
CLIP zero-shot on the catalogue test photos with three ways to write the prompts: the class name, “a photo of a …” (Photo) and eight templates. The axis starts at 0.7. One test photo is 0.006 of accuracy; one plant pot is 0.05 of its recall.
Text description of the chart

A column chart with two groups of three columns, on an axis from 0.7 to 1. Accuracy on the 160 test photos: class name only 0.900, “a photo of a” 0.906, eight templates 0.925. Plant pot recall on its 20 test photos: class name only 0.90, “a photo of a” 0.80, eight templates 0.80.

Many guides say that “a photo of a plant pot” works better than “plant pot”, because CLIP’s training captions are sentences, not single words. Here, that advice did not hold. The overall accuracy rose by one photo (0.900 to 0.906). Plant pot got worse: 18 of 20 right with the name alone, 16 of 20 with “a photo of a”. The eight templates did best overall, with 0.925: four more photos right than the name alone. They did not help plant pot either.

Be careful with these differences. The test set has 160 photos, so one photo moves the accuracy by 0.006. A difference under about 3 points can be chance. The eight templates win by 2.5 points: a small, real gain at best. The lesson is not “this wording is best”. The lesson is: measure the wording on your own photos, and prefer a prompt set to one sentence.

With “a photo of a {class}.”, these are the mistakes among the pot classes:

Five photos with their true label and CLIP's answer. Sunflowers in a white bucket: plant pot, said vase. A wire basket with tulips and painted eggs: plant pot, said vase. A glass bowl of succulents seen from above: plant pot, said bowl. A painted ceramic pot with a plant on a dark table: plant pot, said bowl. A white cracked jug with a pink flower and plants around it: vase, said plant pot.
The pot-class test photos that CLIP zero-shot ("a photo of a …") got wrong: the true label, then CLIP's answer. Photos: mk97007, Nicolas Huk, ProFlowers, Pat (Cletch) Williams, ERIC FORGET, CC BY 2.0, from Open Images.
Text description of the diagram

Five photos. Sunflowers in a white bucket, labelled plant pot: CLIP said vase. A wire basket with tulips and painted eggs, labelled plant pot: CLIP said vase. A round glass bowl of succulents seen from above, labelled plant pot: CLIP said bowl. A painted ceramic pot with a plant, on a dark table, labelled plant pot: CLIP said bowl. A white, cracked jug with a pink flower, among plants, labelled vase: CLIP said plant pot.

Look at them as Hamid’s worker would. Each mistake is a container with two honest names: a glass bowl full of plants, a bucket of flowers, a jug among plants. CLIP’s words come from the web, where people use “vase”, “bowl” and “pot” loosely. Larkfield’s catalogue team uses them in its own way. Zero-shot cannot learn that way: only labelled examples can teach it. That is the job of the next lesson.

Put zero-shot next to the methods of Module 2 in the comparison table. All rows are on the same 160 test photos:

Method Labelled photos Accuracy Worst class
Colour histogram + 5-NN 453 0.263 vase 0.05
ResNet-18, frozen + last layer 453 0.812 candle 0.45
ResNet-18, fine-tuned 4 epochs 453 0.850 candle 0.40
ConvNeXt-Atto, frozen + last layer 453 0.887 candle 0.80
ResNet-50, frozen + last layer 453 0.894 plant pot 0.80
CLIP zero-shot, eight templates 0 0.925 plant pot 0.80

CLIP, with no label at all, is at least as good as every Module 2 method that used all 453 labels. Its lead over ResNet-50 is 5 photos, about 3 points: at the edge of noise. It also does not suffer from the small candle class: it found all 20 test candles, because it never learned from the unbalanced training photos. On the warmer new-studio photos, it gives 0.912, almost the same.

Tomás is delighted: “So we never need to label anything again.” Priya is careful: “Zero-shot knows the web’s meaning of our words. When our meaning is different, as for plant pots, it stays wrong, and we have no way to teach it. And 0.925 means 12 wrong photos in 160.”

Use it:

  • On the first day of a new category, before any photo is labelled.
  • To sort a large pile of photos roughly, so that people label faster: they correct CLIP’s answer instead of starting from nothing.
  • As a baseline that any trained method must beat.

Do not use it alone:

  • When your classes follow your own rules, not everyday words. Plant pot against vase is Larkfield’s rule.
  • For small details. The comparison table has a CLIP prompt for “a damaged tea light candle”. It costs about 1,013 dollars a day on the returns test. That is better than no inspection (2,229), but far from the best method of Module 6. A whole-photo embedding hardly notices a small chip.
  • For words that CLIP never saw in captions, such as Larkfield’s product codes.

Costs: the model is 605 MB and needs about 8 ms per photo on a laptop CPU, in batches. The text embeddings are computed once per class. The weights are under the MIT licence. They learned from web captions, with the web’s mistakes and biases. Module 7 returns to the licences and the risks of such models.

Guided practice: zero-shot in the notebook

Section titled “Guided practice: zero-shot in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The first installs the packages, with open_clip. The second ends with a Ready: line.
  3. Run section 1. It downloads CLIP once. You see (120, 512) (160, 512).
  4. Run the next cell. The live embedding of photo 1 and the stored one have a cosine of 1.0: they are the same model.
  5. In section 2, predict the two closest classes for photo 2. Then run the cell. You see vase 0.563 and plant pot 0.434.
  6. Run section 3. You see the accuracies 0.9, 0.906 and 0.925, and plant pot recall 0.9, 0.8 and 0.8.

Open Images, where the catalogue photos come from, calls this class “Flowerpot”. Maybe CLIP knows that name better. In section 4 of the notebook, replace “plant pot” with “flowerpot” in the class names. Then measure the accuracy with the eight templates on the validation photos, and put it in flowerpot_val. Before you run it, predict: will “flowerpot” win?

Show the answer
names = NAMES.copy()
names[CLASSES.index("plant_pot")] = "flowerpot"
flowerpot_val = round(float(np.mean(np.array(zero_shot(Xval, names, EIGHT)) == val["label"].values)), 3)
print(flowerpot_val)
0.967

The next cell measures “plant pot” on the same validation photos: also 0.967. Both names get 116 of the 120 validation photos right. The validation set cannot choose between them. So you keep the simpler choice, the name your team uses, and you do not look at the test set to break the tie. The failure case below shows why.

Common mistake

Choosing the prompt on the test set

Symptom: Tomás tries five names for the plant pot class with the eight templates, and scores each on the 160 test photos:

Name for plant pot Test accuracy Validation accuracy
plant pot 0.925 0.967
flowerpot 0.938 0.967
flower pot 0.938 0.950
pot plant 0.944 0.967
houseplant in a pot 0.931 0.958

He keeps “pot plant” and reports 0.944 to Omar: “Two points better than the course’s prompts.”

Cause: he chose the name because of its test score. Some of that gain is luck: these 160 photos happen to suit “pot plant”. The best of five tries is always a little too good, as Train, validation and test sets showed with 100 random rules. The test score is no longer an honest estimate for new photos. On the validation photos, the five names are within one or two photos of each other: there, no name is clearly better.

Fix: compare prompts on the validation photos only. When validation cannot tell them apart, keep the simplest prompt, and do not break the tie with the test set. Score the final choice on the test set once. To find the best name, label more validation photos first. 120 photos cannot show a difference of one or two photos.

Consequence in AI work: prompt wording feels free to change, so teams change it often, and they check each change on the same small set. After twenty changes, the score on that set says more about the set than about the prompt. A prompt is a setting of the model, like any other: choose it on validation, report it on test.

Check your understanding

Explain in your own words how CLIP classifies a photo into one of eight classes with no labelled photo. Why is this called zero-shot?

Show the answer

CLIP has an image encoder and a text encoder that put photos and sentences in the same space. It learned this from about two billion photos with captions. In training, each photo gets a high cosine with its own caption and a low cosine with the others. To classify, you write one prompt per class, such as “a photo of a vase.”. You embed the eight prompts and the photo, and take the class whose prompt has the highest cosine. It is zero-shot because the classifier saw zero labelled examples of Larkfield’s classes: only their names.

Check your understanding

Photo 2 (sunflowers in a bucket) gets vase 0.563 and plant pot 0.434. Tomás says: 'CLIP is 56% sure that it is a vase.' What would you tell him?

Show the answer

The two cosines are 0.244 and 0.241: almost equal. CLIP multiplies them by 100 before the softmax, so a difference of 0.003 in cosine becomes 0.56 against 0.43. The numbers mean “a close call between vase and plant pot”, not a measured chance of 56% of being right. They are not calibrated probabilities. The photo is also honestly ambiguous: cut flowers in a bucket are used like a vase. Such a photo should go to a person, or to the labelling guide.

Check your understanding

Omar adds a new category, garden lanterns, tomorrow. No photo is labelled yet. What would you do on day 1 with CLIP, and why would you not stop there?

Show the answer

On day 1, add a prompt set for “garden lantern” (several templates), and use CLIP zero-shot to sort the new photos roughly. People can then correct its answers, which is faster than labelling from nothing. Measure it on a small validation set as soon as one exists. Do not stop there: zero-shot follows the web’s meaning of the words. If Larkfield’s lanterns overlap with its lamps or candles in Larkfield’s own way, CLIP cannot learn that rule. Also, its accuracy here (0.925) still means 12 wrong photos in 160. With a few labelled photos per class, a small trained classifier can learn Larkfield’s line: the next lesson.

Recap

  • CLIP embeds photos and sentences in one space. It learned this with contrastive training on about two billion photo–caption pairs.
  • Zero-shot: write one prompt per class, and take the prompt with the highest cosine. No labelled photo is needed.
  • CLIP’s cosines are small (about 0.1 to 0.3). The factor 100 and the softmax turn small differences into large probabilities: read them as “clear” or “close call”, not as calibrated chances.
  • On Larkfield’s test photos: class name only 0.900, “a photo of a” 0.906, eight templates 0.925. “A photo of a plant pot” did not beat “plant pot” here (plant pot recall 0.90 → 0.80). Measure the wording on your own photos.
  • With zero labels, CLIP is at least as good as every Module 2 method that used 453 labels (best 0.894). Its mistakes are containers that Larkfield names in its own way.
  • Choose prompts on the validation photos, never on the test set.

You can now classify with no labels at all, and with all of them. But zero-shot cannot learn Larkfield’s own rules, and full training needs many labels. Next, you try the cheap middle way: a small classifier on embeddings, with 5, 20 or all photos per class.

Next: Linear probes and few-shot