Meet the Larkfield images
- Lesson 4
- 35 min
- Module 1: Seeing is a business problem
Where we are
We know that an image is a grid of numbers, and that light and size change them. Now we open the real images. In this lesson, we run the setup notebook, look at each Larkfield image set and its licences, and meet the comparison table that every module fills. First, a cook checks her kitchen before she starts.
Lesson 4 of 30 · Vision AI: See, Count and InspectMeet the Larkfield images
1. Seeing is a business problem
2. Baselines and pretrained classifiers
3. Embeddings, CLIP and few labels
4. Detect and count
6. Inspect for defects
7. Match, search and describe
8. Run vision in production
What you will be able to do
Open every Larkfield image set, check that your setup works, and read the comparison table that the whole course fills. A method is only as good as the images under it and the way you score it.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson, with the instructions. Its setup cells install NumPy, pandas, Pillow, scikit-learn, scikit-image and matplotlib. They download the course’s images (about 38 MB, as zip files), so the notebook needs no local setup. About 25 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The two mistakes and their costs, from The return that cost twice.
- The five kinds of answer, from Five kinds of answer, and pixels and channels, from Images are numbers.
- Train, validation and test sets, from Train, validation and test sets in Machine Learning: From Problem to Reliable Model.
- How to run a cell in a notebook and read a pandas table. The machine learning course uses both from its first lessons.
What you need on your computer
- The notebook route (recommended): a browser and a free Google account for Colab, or a free Kaggle account. Nothing to install.
- The project route (for the second half of this lesson): Python 3.12, Git and a terminal. The steps are in The project on your computer below.
Tested with Python 3.12.13 on macOS, in Jupyter and in a terminal. The Windows commands, Linux, Colab and Kaggle were not tested.
Check the kitchen before you cook
Section titled “Check the kitchen before you cook”A cook in a restaurant starts her shift without cooking anything. She reads the orders of the day. She checks that every ingredient is there and fresh, and she puts each one in its place. None of this is cooking. But if the flour is old or the salt is missing, every dish of the evening is wrong. She finds out only when a guest complains.
A vision method is the same. If one image set is missing, or a label is wrong, the method still prints a number. Nothing raises an error. The number is just wrong. So before the course builds any method, you check the images and the scoreboard once, carefully. That is this lesson.
Open the notebook and run the setup
Section titled “Open the notebook and run the setup”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the first setup cell. It installs the packages and prints one line per package, for example
numpy 2.5.4. - If you see other versions, restart the kernel, then run the cell again.
- Run the second setup cell. It downloads five zip files into a folder
larkfield-images, checks each file’s checksum and unpacks them intolarkfield-images/data. - Check its last line:
Ready: 2113 files in larkfield-images/data. - Run the helpers cell. It loads NumPy as
np, pandas aspd, Pillow’sImage, matplotlib and the course’s helpercompare.pyascp. It prints nothing.
A checksum is a short fingerprint of a file. If one byte of a download is wrong, the fingerprint is different, and the cell stops with wrong checksum. Then delete the folder larkfield-images, and run the cell again. If the cell says Could not download, check your internet connection. In Kaggle, also turn on Internet in the notebook settings.
The Larkfield image sets
Section titled “The Larkfield image sets”The course has four image sets and a few changed copies of them. Each set belongs to one kind of answer, and to the people at Larkfield who ask for it:
Text description of the diagram
The diagram has five boxes in a row (one under the other on a phone). The catalogue has 733 product photos in 8 categories, 256 × 256 pixels: 453 for training, 120 for validation and 160 for testing. They are cut from Open Images photos under CC BY 2.0, and each photo’s author is listed. Modules 2, 3, 7 and 8 use them. The returns have 735 tea light photos: 380 good and 25 damaged for training, 100 and 12 for validation, 200 and 18 for testing. They are VisA photos under CC BY 4.0, used in Modules 5 to 8. The pallets are 90 scenes of 512 × 384 pixels, drawn by our code (CC0, synthetic): 40 for training, 20 for validation and 30 for testing, used in Modules 4, 5 and 7. The customer photos are 160 phone photos of the test products, made from them by code, used in Module 7. The changes are the new studio, 160 test photos with warmer light, and a month of 135 returns photos with no labels. Our code changed both, and Module 8 uses them. Under the five boxes, a dashed box lists the hidden test sets: 160 catalogue photos, 135 returns and 30 pallets. They are not in the course files. The course team keeps them and uses each one once, to confirm a choice made on the validation set.
The catalogue
Section titled “The catalogue”Section 1 of the notebook reads the catalogue’s manifest: a table with one row for each photo. It gives the file, the split (train, val or test), the label, and where the photo comes from:
catalogue = pd.read_csv("data/catalogue/manifest.csv")print(catalogue["split"].value_counts().to_string())print(catalogue[catalogue["split"] == "train"]["label"].value_counts().to_string())The output:
splittrain 453test 160val 120labelchair 70vase 70lamp 67plant_pot 66mug 63bowl 61cushion 41candle 15The catalogue has 733 photos, and the eight categories do not have the same number of training photos. The chart makes one of them stand out:
- Training photos
- Candle: a problem we planted
Text description of the chart
A column chart with eight columns, one per category, on an axis from 0 to 80 photos. Bowl has 61 training photos, candle 15, chair 70, cushion 41, lamp 67, mug 63, plant pot 66 and vase 70. The candle column is much shorter than all the others.
Candle has only 15 training photos. This is a problem we planted: Larkfield started to sell candles only recently. A method that learns from many examples of the other categories may ignore the small one. Module 3 measures this and fixes it.
Real labels are also noisy. The catalogue photos come from Open Images, a large public set of photos with labels that people checked. Even so, some “lamp” photos are carved pumpkins with a light inside, and some “mug” photos are beer glasses. The course keeps them, because real catalogues have such photos too. Section 7 of the notebook shows some of the lamp photos: look at them and ask whether your model should learn from a pumpkin.
Licences and authors
Section titled “Licences and authors”Every catalogue photo has an author and a licence in the manifest. The photos are under CC BY 2.0. This licence lets you use, change and share the photo, also in a business, if you name the author and the licence. So every catalogue photo on this course’s pages has its author in the caption, and you should do the same in your own work.
Open Images says that it cannot guarantee the licence of every photo. For a real product, Larkfield would use its own photos. For learning, open photos with a named author are a good choice.
No catalogue photo shows a person. The course removed 160 photos with a person in them. It used four checks: the people labels of Open Images, a face detector, a person detector, and a person who looked at every photo. If you still find a person, tell us. The course does not build anything that recognises people. The image service of Module 8 blurs faces by default.
Returns, pallets and customer photos
Section titled “Returns, pallets and customer photos”Section 3 reads the returns manifest and counts the photos by split and status:
returns = pd.read_csv("data/returns/manifest.csv")print(returns.groupby(["split", "status"]).size().unstack().to_string())status damaged goodsplittest 18 200train 25 380val 12 100These are the tea light photos of The return that cost twice: 18 of the 218 test photos are damaged. Each damaged photo has a mask, saved as PNG, as Images are numbers recommended. The photos come from VisA, a public set of inspection photos made by researchers at Amazon, under CC BY 4.0.
The notebook then draws a pallet with its boxes, and shows a customer photo next to its catalogue photo. Both are synthetic. Our code drew the pallets, so every box is known exactly. It also made each customer photo from a catalogue test photo: another angle, poor light, blur and noise. Real phone photos of the same products do not exist as open photos.
A photo as numbers
Section titled “A photo as numbers”Section 2 opens one catalogue photo as numbers, as in Images are numbers:
path = "data/" + catalogue.iloc[0]["file"]pixels = np.asarray(Image.open(path))print(pixels.shape, pixels.dtype)print("the pixel in the middle:", pixels[128, 128], " mean of each channel:", pixels.reshape(-1, 3).mean(0).round(1))(256, 256, 3) uint8the pixel in the middle: [66 61 55] mean of each channel: [137.3 145.7 158.3]The shape is rows, columns and channels. uint8 is the type of each number: a whole number from 0 to 255. The pixel in the middle is dark grey, and the photo is a little blue on average.
The hidden-test rule
Section titled “The hidden-test rule”You know the three sets of the machine learning course. You learn on the training set, choose on the validation set and score once on the test set. This course adds one more rule. For the catalogue, the returns and the pallets, the course team keeps a hidden test set. It is not in the course files, and nobody chooses anything with it.
Why a fourth set? The test set is in your files. Every person who tries “just one more change” on it uses it a little, often without meaning to. The hidden test set cannot be used this way. Twice in the course, a lesson shows what happened when the method chosen on the validation set met its hidden test, once. One of those results is a real surprise, and Module 6 explains it.
The course also has a file data/sealed-key.txt. It holds the answers to two planted problems that you search for in Module 3: wrong labels and photos that are in the catalogue twice. The file is packed so that you do not read it by chance. It is sealed, not secret: compare.reveal() unpacks it to score your search. Treat it like the answers at the back of a book.
The comparison table
Section titled “The comparison table”Every module runs its methods on the same photos, with the same splits, and adds rows to one comparison table. This is how a colour rule of Module 2 and a large model of Module 7 can be compared fairly. Section 4 of the notebook prints the cost table and the first part of the comparison table:
print(json.load(open("data/costs.json")))table = cp.Table.load("data/comparison_table.json")table.show("catalogue", columns=("method", "setting", "value", "worst", "worst_value", "model_mb", "labelled_images"))The first lines of the table:
catalogue: accuracymethod setting value worst worst_value model_mb labelled_imagescolour and brightness rules catalogue test 0.269 bowl 0.000 0 0colour and brightness rules new studio 0.244 bowl 0.000 0 0colour histogram + 5-NN catalogue test 0.263 vase 0.050 0 453colour histogram + 5-NN new studio 0.225 cushion 0.000 0 453HOG + logistic regression catalogue test 0.356 candle 0.100 0 453HOG + logistic regression new studio 0.356 candle 0.100 0 453How to read a row
Section titled “How to read a row”| Column | What it says |
|---|---|
task |
The kind of job: catalogue (labels), inspect (returns), count (boxes), match or attributes (text) |
method, setting |
What was run, and on which photos: for example the catalogue test or the new studio |
value |
The task’s main number: accuracy for the catalogue, cost a day for inspection, count error for pallets |
worst, worst_value |
The category with the worst result, and that result |
ms_per_image |
CPU time for one photo, in milliseconds |
model_mb |
The size of the model file, in megabytes |
labelled_images |
How many labelled photos the method needed |
licence |
The licence of the model’s weights, or “our code” |
The table has 66 rows in five tasks: 37 for the catalogue, 12 for inspection, 10 for counting, 5 for matching and 2 for attributes. The tasks cannot share one number, as Five kinds of answer showed. So each task has its own value, and you compare rows only within a task.
Read the first rows with what you know. The colour rules and the colour histogram fall under the new studio light; HOG does not move. The worst column tells more than the main number: the colour rules get no bowl right (0.000).
One more caution. The catalogue test has 160 photos, so one photo is 0.6 points of accuracy. Treat a difference under about 3 points between two methods as noise.
The project on your computer
Section titled “The project on your computer”The notebook is for exploring. The course also has a small Python project, larkvision, with a command-line tool called vision. It grows in each module, and in Module 8 it becomes a small image service. Today you install its first version, the snapshot start, and run its three commands.
-
Open a terminal, and go to the folder where you keep your projects, for example
~/projects. -
Get the course’s practice files:
Terminal window git clone https://github.com/Nextia-Intelligence/nextia-learning-labs.gitWithout Git, download the ZIP file of the repository and unzip it here. Its folder is then
nextia-learning-labs-main: use that name in the next steps. -
Copy the snapshot
startto a new folderlarkvision, besidenextia-learning-labs, and go into it:Terminal window cp -R nextia-learning-labs/C42/snapshots/start larkvisioncd larkvisionTerminal window Copy-Item -Recurse nextia-learning-labs\C42\snapshots\start larkvisioncd larkvision -
Create a virtual environment, and install the project. A virtual environment is a folder with its own Python packages, so this project does not change your other projects:
Terminal window python3.12 -m venv .venvsource .venv/bin/activatepip install -r requirements.txtpip install -e . --no-depsTerminal window py -3.12 -m venv .venv.venv\Scripts\Activate.ps1pip install -r requirements.txtpip install -e . --no-deps -
Tell the
visioncommand where the course’s images are. The environment variableLARKVISION_DATAholds the full path of the foldernextia-learning-labs/C42/data. Set it in the same terminal:Terminal window export LARKVISION_DATA="$(cd ../nextia-learning-labs/C42/data && pwd)"Terminal window $env:LARKVISION_DATA = (Resolve-Path ..\nextia-learning-labs\C42\data).Path -
Run the tests. You see
9 passed:Terminal window python -m pytest -q -
Run
vision info. It describes the data folder, with the licences and the costs.
An environment variable lives only in the terminal where you set it. So each time you open a new terminal for this course, go to larkvision, activate .venv and set LARKVISION_DATA again. These are steps 4 and 5, without the two pip commands. The images stay in one place, in nextia-learning-labs, and every module of the course reads them from there.
The output of vision info proves that the setup works. Its first line shows the data folder on your own computer:
Data folder: /Users/you/projects/nextia-learning-labs/C42/dataCatalogue: 453 train, 120 validation, 160 test photos in 8 categories; new studio: 160 (light changed by code)Returns: 405 train (25 damaged), 112 validation (12 damaged), 218 test (18 damaged); month: 135 photos, no labelsPallets: 40 train, 20 validation, 30 test photos (drawn by code)Customer photos: 160 (simulated from catalogue photos by code)Licences: 870 photos CC BY 4.0; 733 photos CC BY 2.0; pallets CC0 1.0Costs (assumptions): damaged item sold as new 45 dollars, good item marked damaged 28, a person's check 0.50; 100 checks a day for 600 returnsThe hidden test sets are not here: the course team keeps them.The 870 photos under CC BY 4.0 are the VisA returns: 735 with labels and the 135 of the month. Now print the inspection rows of the comparison table:
vision compare --task inspectNear the end of the output are three reference rows. You priced them in The return that cost twice. Here they are, with the columns on the right cut:
no inspection (every return goes back on sale) returns test 2229.4 all damaged items sold -a person checks 100 random returns a day returns test 1907.8 expected value -a perfect inspector checks only the damaged items returns test 24.8 the floor -The other rows are methods that later modules build, from a colour rule to models that compare small parts of each photo. Every one of them costs between 24.8 and 2,229.4 dollars a day. Last, run vision reset. It removes your own outputs and checks every data file against its checksum:
Reset: removed outputs; data: all 2113 files are right.It says removed outputs because vision compare made an empty outputs folder for your own rows. If you ever change a data file by mistake, vision reset names it, for example changed: catalogue/train/candle/P-00402f5d.jpg. To repair it, run git checkout -- C42/data in the folder nextia-learning-labs, or unzip the ZIP file again.
Common mistake
No data folder found
Symptom: the next day, you open a new terminal, go to larkvision, activate the environment and run vision info. You get this, and nothing else:
Error: no data folder found (set LARKVISION_DATA, or copy the course's data folder here)Cause: the new terminal does not have LARKVISION_DATA: an environment variable lives only in the terminal where you set it. Without it, the vision command looks only in data, ../data and ../../data, from the folder where you are, and the images are not there.
Fix: set the variable again, from the folder larkvision, as in step 5 above. Then run vision info again: its first line names the data folder. If the error stays, check that nextia-learning-labs/C42/data/catalogue/manifest.csv exists beside larkvision.
Consequence in AI work: a tool that guesses where its data is can also find the wrong data: an old copy or another set. The error message is the good case. Before you trust a number, check which data folder made it: vision info prints it on its first line.
Your turn: the damaged training photos
Section titled “Your turn: the damaged training photos”Section 6 of the notebook has one task with a check cell. Each damaged photo has one or more damage types in the column damage, with "; " between two types.
- Take the training photos of the returns whose status is
damaged. - Split each
damagetext at"; ". - Count each type. Put the result in
damage_counts: a dictionary from type to count. - Run the check cell. It prints
Check passed: damage_counts is right.when your answer is right.
Show the answer
from collections import Counterdamage_counts = Counter()for text in returns[(returns["split"] == "train") & (returns["status"] == "damaged")]["damage"]: for kind in text.split("; "): damage_counts[kind] += 1damage_counts = dict(damage_counts)print(damage_counts){'foreign particals on candle': 6, 'damaged corner of packaging': 7, 'weird candle wick': 3, 'different colour spot': 6, 'chunk of wax missing': 5, 'wax melded out of the candle': 2, 'extra wax in candle': 2}The 25 damaged training photos have 31 damage types: some photos have two. Seven types share 25 photos, so each type has only 2 to 7 examples. A method that must learn each damage type from labelled examples has very little to learn from. Module 6 starts from this fact. The type names come from VisA as they are, with their spelling (“particals”, “melded”).
If something breaks in the notebook, delete the folder larkfield-images and run the notebook again from the top.
Knowledge check
Section titled “Knowledge check”Check your understanding
The candle category has 15 training photos, and the other categories have 41 to 70. Why can this matter for a model, and why does the course keep it?
Show the answer
A model that learns from examples sees candles much less often than chairs or vases. It can learn to prefer the large categories and get candles wrong, while its overall accuracy still looks good. The worst column of the comparison table shows this. In the first rows, candle is the worst category for HOG, at 0.100. The course keeps the small category because real catalogues have new categories with few photos. It is a problem we planted, and Module 3 measures it and fixes it.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why does the course keep hidden test sets, when you already have a test set in your files?
Show the answer
The test set is in your files, so each look at it, and each change made after a look, uses it a little. Its score then becomes too good, like an exam paper seen the evening before. A hidden test set is not in the files, so nobody can choose anything with it. When a lesson scores a method chosen on the validation set on the hidden test, once, the result is an honest estimate for new photos.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
In the comparison table, a catalogue method has a value of 0.894 and an inspection method a value of 369. Which one is better? What do you compare instead?
Show the answer
Neither: the two numbers measure different things. For the catalogue, value is an accuracy, where higher is better. For inspection, it is a cost in dollars a day, where lower is better. Compare rows only within one task. Within a task, also look at the worst column, the CPU time, the model size, the labelled photos needed and the licence. With 160 test photos, treat differences under about 3 points of accuracy as noise.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- The course has four image sets. The catalogue has 733 photos (CC BY 2.0, each with its author), and the returns 735 VisA tea light photos (CC BY 4.0). The pallets (90 scenes) and the customer photos are made by our code. The new studio and the month are changed copies.
- Some problems are planted: candle has only 15 training photos, and some labels are wrong. No catalogue photo shows a person: 160 were removed.
- The hidden test sets are not in your files. They are used once, for a method chosen on the validation set.
sealed-key.txtis sealed, not secret. - The comparison table has 66 rows in five tasks. Each task has its own main number; compare rows only within a task, and treat differences under about 3 points as noise.
- The
larkvisionproject sits besidenextia-learning-labsand finds the images throughLARKVISION_DATA. Its first snapshot runsvision info,vision compareandvision reset. The reference rows for returns are 2,229.4, 1,907.8 and 24.8 dollars a day.
You can now load every image set and read the comparison table. But the table’s first rows are only numbers that someone else measured. Next, you build the simplest methods yourself, colour rules and colour histograms, and see where they win and where they break.