When not to build it
- Lesson 26
- 25 min
- Module 7: Match, search and describe
Where we are
In this course, we have built rules, classifiers, embeddings, detectors and vision-language models, and measured each one on the same photos. Sometimes the best answer is none of them. In this lesson, we look at problems that a better camera, a barcode, fixed light or a person solve more cheaply, and at tasks that are not ready for a model. First, a dark staircase.
Lesson 26 of 30 · Vision AI: See, Count and InspectWhen not to build it
1. Seeing is a business problem
2. Baselines and pretrained classifiers
3. Embeddings, CLIP and few labels
4. Detect and count
6. Inspect for defects
7. Match, search and describe
8. Run vision in production
What you will be able to do
Before you build a vision model, check whether a cheaper change solves the problem: a barcode, a better camera, fixed light, or a person. Also check whether the task is ready for a model at all.
Before you startWhat to know and to have ready
What you need to know
- The costs of Hamid’s returns, from The return that cost twice.
- The lines that did not hold on the hidden test, from Set the line with the warehouse.
- Matching and the vision-language results, from the three earlier lessons of this module.
What you need on your computer
Nothing. This lesson has no code and no notebook. Your larkvision project stays at the snapshot end-of-m07.
The dark staircase
Section titled “The dark staircase”The stairs to a warehouse’s basement are dark, and workers trip on them. Someone suggests a camera with a model that sees a person fall and calls for help. Someone else buys a light bulb and paints the edge of each step yellow. Nobody trips again.
The model would have worked, in a way: it would have seen the falls. But it answered the wrong question. The question was not “how do we see a fall?” It was “how do we stop the falls?” The cheaper fix changed the place, not the picture of it.
The problem: a model for everything
Section titled “The problem: a model for everything”Tomás has a plan. “The hosted vision-language model reads photos, counts boxes and finds damage. Let us use it for everything.” This module and the ones before it measured what that would mean:
- Damage on returns. The hosted model called 17 of 22 good candles damaged. If that rate held for Hamid’s 600 returns, it would cost about 11,910 dollars a day (an estimate). No inspection at all costs 2,229.4.
- Product types. On the same 50 photos, the CLIP probe was right on 48, and the hosted model on 42.
- Matching. DINOv2 found the right product first for 0.944 of the customer photos. That still leaves 9 of 160 first answers wrong.
- Small damage. The smallest planted damage, a colour spot, is 30 pixels in the camera’s full photo and 0.91 pixels at 224 pixels wide. No model finds a spot that is smaller than one pixel.
Each of these problems has a cheaper fix than a bigger model. Some have a fix that needs no model at all.
The idea: try the cheaper fix first
Section titled “The idea: try the cheaper fix first”Before you build a model, ask four questions, in this order:
Text description of the diagram
The diagram has four questions in order (one under the other on a phone). First: can a barcode, an order number or a form answer it? If yes, use it, with no model. If no, the second question: can you fix the camera, the light or the place of the item? If yes, fix it first and measure again. If no, the third question: is there a fallback, meaning a person and a budget for the unsure cases? If no, the task is not ready: plan the fallback first. If yes, the fourth question: are there labels good enough to measure the model? If no, the task is not ready: label a test set first. If yes, build the smallest method that works, and measure it against the cheaper fix.
The first two questions look for a cheaper fix. A cheaper fix removes the problem, so that no model or only a simple one is needed. The last two questions check that the task is ready. A model without a fallback, or without labels to measure it, is a risk at any accuracy.
Here are three Larkfield problems from this course, each with its cheaper fix:

Text description of the diagram
The first panel shows a small part of a damaged tea light, enlarged so that each pixel is a square. At 320 pixels wide, the colour spot is a small dark mark of about 3.4 pixels. At 224 pixels wide, it shrinks to about 1.64 pixels and almost disappears. The fix is a better camera, or a closer one. The second panel shows the same white bowl with a black spoon under the studio light and under a warmer light, which our code added. The fix is fixed light. The third panel shows two white bowls that CLIP confused: the one with the black spoon, and a plain one seen from above. The fix is a barcode.
A barcode, an order number or a form
Section titled “A barcode, an order number or a form”Most returns arrive with a return form and an order number. The order says which product it is, exactly, with no photo and no model. Matching a photo to the catalogue is useful only when nothing else says which product it is. An example is a customer’s email with a photo and no order number. Even then, the support team can ask for the order number before it searches.
The same holds for every number that is printed. If the answer is written on the item or its paper, read it, do not guess it from the picture. A barcode scanner reads the exact product number. Text on a label, a receipt or a return form needs OCR: optical character recognition, which reads printed text from an image. That is a different problem. Document and Multimodal AI teaches it: reading receipts, scans and forms, and showing where every value came from.
A better camera, fixed light, a fixed place
Section titled “A better camera, fixed light, a fixed place”Many vision problems start at the camera. The colour spot in the first panel has 54 pixels in the camera’s full photo. The course’s photos are 320 pixels wide, so it has about 3.4, and a model that works at 224 pixels sees about 1.64. A bigger model cannot see pixels that are not there. A camera closer to the candle, or a model that works on parts of the full photo, can.
Light is the same. Under the new studio’s warmer light, the colour histogram fell from 0.263 to 0.225 on the catalogue test, with no other change. A fixed lamp with a known colour costs little. So does a grey card in the corner of each photo. Both remove the problem for every method. A fixed place for the item helps too: a mark on the table where every candle stands. With a fixed camera, fixed light and a fixed place, even the colour rules of Module 2 become useful again, as that module showed.
A person and a budget
Section titled “A person and a budget”Hamid’s team checks 100 returns a day, at 0.50 dollars a check: 50 dollars a day. That is the fallback: the place where unsure cases go. A model without a fallback must decide every case alone, including the ones it gets wrong.
The best inspector of Module 6 cost 369 dollars a day on the test set. On the hidden test, the same lines cost 1,468.9 dollars a day, because the lines were set on only 12 damaged photos. In real use, there is no hidden test. Only the team’s checks on a sample of passed items would show such a drop. The person is not a sign that the model failed. The person is part of the system.
Labels to measure it
Section titled “Labels to measure it”The last lesson measured three attributes on 50 photos labelled by one person. No field could show 95%. A task with too few labels, or labels that one person made alone, is not ready for a model that decides alone. It is ready for a model that helps people label. The first investment is a good test set, not a bigger model.
Tasks that are not ready for a model
Section titled “Tasks that are not ready for a model”Three signs say: wait, and fix the task first.
- Low-quality labels. If two people would label the same photo differently, the model learns the disagreement. The catalogue’s “lamps” include a glass furnace and carved pumpkins. Fix the labelling rule first.
- An unstable camera. If the camera, its position or the light changes without warning, every measured number can become wrong overnight. Module 8 shows a camera change that makes every photo fail. Fix the camera, or plan to measure again after every change.
- No fallback. If nobody can check the unsure cases, there is no safe way to use a model’s confidence. Plan the review budget first.
The licence is part of “ready”
Section titled “The licence is part of “ready””One more check: may Larkfield use the model in a product? Each model’s weights have a licence, and the data it learned from has its own terms. The timm ResNet and ConvNeXt weights of this course are Apache-2.0, but they learned from ImageNet. ImageNet’s terms of access, item 1, say: “Researcher shall use the Database only for non-commercial research and educational purposes.” So “Apache-2.0 weights” does not settle whether a shop may use them. The same question applies to every pretrained model, and to the code around it: The YOLO family and its licence shows a popular detector whose licence asks you to share your source code. Before a model goes into a product, write down its licence, its training data and the data’s terms. Then ask the person in your company who decides on licences.
Choose with cost, accuracy and risk
Section titled “Choose with cost, accuracy and risk”This is the module’s main decision. For each request, compare the three families of this course on the same three questions.
| Larkfield request | CLIP (zero-shot or a probe) | A model trained on our photos | A vision-language model |
|---|---|---|---|
| Sort new catalogue photos into 8 types | Best, by a small margin. Probe 48 of 50; zero-shot 0.925 with no labels; about 8 ms a photo | 0.850 (ResNet-18 fine-tuned), 0.894 (ResNet-50 frozen): close to CLIP, within the noise of 160 photos; needs 453 labels | 42 of 50; 1.38 dollars per 1,000; photos leave the computer |
| Alt text for 733 photos | Cannot write text | Cannot write text | Best. A first draft for a person to correct, about 1 dollar in all |
| Damaged candles at 600 a day | Weak: CLIP prompt cost 1,013 dollars a day | Supervised ResNet-18: 1,106 a day; PatchCore (no damage labels): 369 on test | Calls 17 of 22 good candles damaged: about 11,900 a day (an estimate) |
| Which product is in a returned parcel | Finds the kind, not the item | Not needed | Not needed: read the order number or the barcode |
The table has no single winner. CLIP wins where a fixed list of classes and fast answers are enough. A trained model wins where you have labels and a measured cost per day. A vision-language model wins only where the answer must be words. And for the last request, the best answer is no model.
Common mistake
The model that fixed the wrong problem
Symptom: after the studio changed its lamps, the catalogue classifier’s accuracy falls. Tomás spends two days on a fix. He adds colour changes to the training photos so that the model learns to ignore the light. The accuracy moves by less than the noise of 160 test photos.
Cause: the problem was the light, not the model. In Module 2, colour augmentation gave 0.863 against 0.850 without it: about two photos, within the noise. The studio’s camera had no fixed white balance. White balance is the camera setting that makes a white object look white under any lamp.
Fix: fix the camera first. Set a fixed white balance, or put a grey card in each photo and correct the colours from it. A simple correction of this kind, the grey-world rule, raised the colour histogram from 0.225 to 0.281 on the new studio photos. Module 8 measures this. Then measure the model again on the same photos.
Consequence in AI work: when a model gets worse, first ask what changed in the world. It can be the camera, the light, the products or the people who take the photos. A change in the world is often cheaper to fix in the world.
Your turn: one request, three families
Section titled “Your turn: one request, three families”Omar adds a new product category, garden lanterns, with about 100 photos. He wants each one sorted into its category, and he wants alt text. Hamid also asks whether the hosted model can find damaged lanterns in returns. For each of the three requests, choose between CLIP, a model trained on Larkfield’s photos and a vision-language model, or no model. Give the cost, the expected accuracy and the main risk. Use the numbers of this module.
Show an answer
Sort into categories: a CLIP probe, after adding about 20 labelled lantern photos. Expected: close to the 0.929 of 20 photos per class in Linear probes and few-shot, measured again on a test set with lanterns. Cost: milliseconds a photo on a CPU, no money per photo. Risk: lanterns look like lamps and candle holders; check the worst class, and send unsure photos to a person.
Alt text: the hosted vision-language model, as a first draft. Cost: about 1.38 dollars per 1,000 photos, so under 1 dollar here. Accuracy: unknown for lanterns; the catalogue fields were 84–93% right on 50 photos. Risk: invented details; a person reads and corrects every text. Send only photos without people.
Damaged lanterns: no model yet. There are no labelled damaged lanterns, and the hosted model’s false alarms on candles would cost far more than no inspection. First, collect good lantern photos with a fixed camera and fixed light, keep Hamid’s team as the fallback, and label a test set. Then try PatchCore, which needs only good photos, and set its lines with Hamid.
Knowledge check
Section titled “Knowledge check”Check your understanding
The smallest planted damage is 30 pixels in the camera's full photo and 0.91 pixels at 224 pixels wide. Tomás suggests a bigger model. What do you suggest instead, and why?
Show the answer
A better or closer camera, or a method that works on parts of the full photo, not a bigger model. At 224 pixels wide, the spot covers less than one pixel. Its colour is mixed with the wax around it, so the information is gone before any model sees it. A bigger model cannot recover pixels that are not there. Fixing the input helps every method, including the simple ones.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Name the four questions to ask before you build a vision model, and give one Larkfield example where the answer to the first question is yes.
Show the answer
- Can a barcode, an order number or a form answer it? 2. Can you fix the camera, the light or the place of the item? 3. Is there a fallback, a person with a budget for the unsure cases? 4. Are there labels good enough to measure the model? Example for the first: which product is in a returned parcel. The return form and its order number say it exactly, so no photo matching is needed. If the answer is printed text, use OCR, as the document course teaches, not a guess from the picture.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The timm ResNet-18 weights used in this course are Apache-2.0. Does that mean Larkfield may use them in its product? What would you check?
Show the answer
Not by itself. The Apache-2.0 licence covers the weights file. The weights learned from ImageNet, whose terms of access allow non-commercial research and education. Whether a shop may use weights trained on that data is a question for the person who decides on licences in the company. Write down, for each model, its licence, its training data and the data’s terms. Then ask before the model goes into a product.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Before you build, ask four questions: a barcode or form? A camera, light or place to fix? A fallback with a budget? Labels to measure it? The first two find a cheaper fix. The last two check that the task is ready.
- Printed answers are read, not guessed: a barcode scanner, or OCR from Document and Multimodal AI.
- A damage of 0.91 pixels at 224 needs a better camera, not a bigger model. A light change needs fixed light or a grey card.
- A person is part of the system: in real use, only the team’s checks on a sample of passed items show lines that do not hold.
- “Apache-2.0 weights” does not settle product use: check the training data’s terms too.
- No single family wins. Use CLIP for fixed classes, and a trained model where labels and a measured cost exist. Use a vision-language model where the answer must be words, and no model where an order number answers.
You can now decide whether to build a model, and which one. But every number so far came from photos taken under the conditions of the training set. Next, the camera, the light or the products change, and you find out how to see the change before Hamid’s costs do.
Next: Domain shift