Classify clothing images

What you will be able to do

Build an image classifier from raw files to a tested, saved model that you can show to others. You will see why a network that reads pixels in a row misses what a picture is, and fix it with a small convolutional network. You will also find out where it fails, test an augmentation honestly, and use the model on new images.

Case studyPortfolio piece

Dataset

Fashion-MNIST (Zalando Research)

Licence

MIT

Time

About 2 hours

Runs in

Colab, Kaggle or VS Code

Skills used

• Images as tensors
• Logistic-regression baseline
• Multilayer network
• Convolutional network
• Early stopping
• Confusion matrix
• Data augmentation
• Saved model and experiment record

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs every step of this page, with the same code and the same output, plus five tasks with checks. Its setup cells download the data (31 MB) and check it, so it needs no local setup. It runs in about 2 minutes on the course’s test computer; allow up to 10 minutes on a free Colab CPU.

Before you startWhat to know and to have readyAbout 120 minutes

What you need to know

This case study uses Modules 1 to 4. These lessons matter most:

• A tensor’s shape and the batch dimension, from Tensors and shapes, and shape errors, from The forward pass.
• Mini-batches, epochs, model.eval() and the two curves, from Batches, epochs and validation.
• Early stopping and data augmentation, from Overfitting controls.
• Looking at examples and labels first, from Debug the experiment.
• Logistic regression and the confusion matrix, from the machine learning course: Classification intuition and Classification metrics.

What you need on your computer (only for the local route; the notebook needs none of it)

• The ticket-router environment from Set up PyTorch, plus scikit-learn and matplotlib: python -m pip install scikit-learn==1.9.1 matplotlib. The notebook installs them if they are missing.
• The notebook, downloaded from the link above. Its setup cell makes a folder clothing-images and downloads the data into it. To start again, delete that folder.

Tested on macOS with Python 3.12, PyTorch 2.14.1 (CPU, 2 threads) and scikit-learn 1.9.1, in Jupyter from a fresh start. On another computer, the last digits of some numbers can differ.

The situation

Brightloom is an online marketplace for clothes. Sellers upload a photo of each product and choose its category. Many choose the wrong one, or none. A coat filed under “Shirt” sits in the wrong aisle. Customers who look for coats never see it, so it sells late or not at all. Ines, the catalogue lead, asks for help: “Can the photo tell us the category?”

Kwame, the data scientist, proposes a suggestion, not a decision. When the model is sure, the upload page fills in the category, and the seller can still change it. When the model is unsure, the page asks: “Is this a shirt or a coat?” So a wrong suggestion costs a seller a click. A wrong suggestion that the seller does not notice costs sales.

Brightloom is made up for this case study. The photos are real: they come from Fashion-MNIST, a public dataset of product photos from the catalogue of the online shop Zalando.

What you will build

You will build a small convolutional network that reads a product photo and gives a probability for each of ten categories. You will compare it with a baseline and a plain multilayer network, under one split and one metric. Then you will choose a threshold for “fill in”, test the model once, save it with an experiment record, and use it on new photos. At the end, you write it up as a portfolio piece.

The skills come from this course: tensors and shapes, the training loop with batches and evaluation mode, early stopping, and data augmentation. The case study adds one new idea, the convolution. It also brings the confusion matrix and the test set from the machine learning course.

The data

Fashion-MNIST has 60,000 training and 10,000 test images. Each image is 28 × 28 grey pixels and has one of ten labels. The authors made it as a harder replacement for MNIST, a classic set of handwritten digits.

Source and licence

Fashion-MNIST by Zalando Research (Han Xiao, Kashif Rasul and Roland Vollgraf, 2017, arXiv:1708.07747), github.com/zalandoresearch/fashion-mnist, MIT License, Copyright © 2017 Zalando SE. The notebook downloads the four original files and checks their SHA-256; nothing is changed. MNIST digits and KMNIST were also compared. MNIST’s original page had no licence statement, and KMNIST is CC BY-SA, a share-alike licence.

The files are not CSV files. They use a simple format called IDX: a few header bytes, then one byte per pixel, row by row. NumPy can read that directly, so the notebook needs no extra package. Torchvision, PyTorch’s image library, can also download Fashion-MNIST. But it is one more install, and it hides the step this case study teaches: bytes becoming a tensor.

clothing.py

import gzip
import numpy as np

CLASSES = ["T-shirt/top", "Trouser", "Pullover", "Dress", "Coat",
           "Sandal", "Shirt", "Sneaker", "Bag", "Ankle boot"]


def read_images(path):
    """IDX images: 16 header bytes (a file code, count, rows, columns), then one byte per pixel."""
    raw = gzip.open(path).read()
    count, rows, cols = (int.from_bytes(raw[i:i + 4], "big") for i in (4, 8, 12))
    return np.frombuffer(raw, dtype=np.uint8, offset=16).reshape(count, rows, cols)


def read_labels(path):
    """IDX labels: 8 header bytes (a file code, count), then one byte per label."""
    return np.frombuffer(gzip.open(path).read(), dtype=np.uint8, offset=8)


images = read_images("data/train-images-idx3-ubyte.gz")
labels = read_labels("data/train-labels-idx1-ubyte.gz")
test_images = read_images("data/t10k-images-idx3-ubyte.gz")
test_labels = read_labels("data/t10k-labels-idx1-ubyte.gz")
print(images.shape, images.dtype, labels.shape)
print(test_images.shape, test_labels.shape)
print("first image's label:", labels[0], CLASSES[labels[0]])

The output:

(60000, 28, 28) uint8 (60000,)
(10000, 28, 28) (10000,)
first image's label: 9 Ankle boot

What this tells us. int.from_bytes(..., "big") turns 4 header bytes into one number, so the file itself says 60,000 × 28 × 28. np.frombuffer reads the rest as unsigned bytes (uint8, 0 to 255). Each label is a number from 0 to 9, and CLASSES gives its name.

First checks

Before any model, Kwame checks the pixel range, the images per class, and repeats. A test image that is also in the training file would make the test too easy.

clothing.py

print("pixel values from", images.min(), "to", images.max())
print("training images per class:", np.bincount(labels).tolist())
print("test images per class:    ", np.bincount(test_labels).tolist())
train_keys = {img.tobytes() for img in images}
print("repeated images in the training file:", len(images) - len(train_keys))
print("test images also in the training file:", sum(img.tobytes() in train_keys for img in test_images))

The output:

pixel values from 0 to 255
training images per class: [6000, 6000, 6000, 6000, 6000, 6000, 6000, 6000, 6000, 6000]
test images per class:     [1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000]
repeated images in the training file: 0
test images also in the training file: 0

What this tells us. The ten classes are exactly balanced. So a guess of one class is right 10% of the time, and accuracy is a fair measure here. No image is repeated, and the test file is clean.

Look at the images

As Debug the experiment says, look at examples before you train anything. Here is the first image of each class.

The first image of each class in the training file, at their real resolution of 28 × 28 pixels. Each product is centred, in grey, on a black background.

Text description of the chart

Ten small grey images on black, each with its class name below: a T-shirt/top, a pair of trousers, a pullover with long sleeves, a dress, a coat, a sandal, a shirt, a sneaker, a handbag and an ankle boot. Each product fills most of the square and is centred.

Even at this size, a person can name most of them. Some pairs are close, though. A shirt, a T-shirt/top and a pullover can differ only in a collar or a sleeve, a few pixels wide.

Images as tensors

What does the model see? Numbers. Kwame prints an 8 × 8 corner of the first image, an ankle boot, where the toe begins:

clothing.py

print(CLASSES[labels[0]])
print(images[0, 16:24, 0:8])

The output:

Ankle boot
[[  0   0   0   0  18  44  82 107]
 [  0  57 187 208 224 221 224 208]
 [  3 202 228 224 221 211 211 214]
 [ 98 233 198 210 222 229 229 234]
 [ 75 204 212 204 193 205 211 225]
 [ 48 203 183 194 213 197 185 190]
 [  0 122 219 193 179 171 183 196]
 [  0   0  74 189 212 191 175 172]]

An 8 × 8 corner of the first ankle boot (rows 16 to 23, columns 0 to 7): what you see and what the model gets. 0 is black, 255 is white. The outlined pixel, 98, is the tip of the toe.

Text description of the chart

On the left, an 8 by 8 square of grey pixels: black in the top-left and bottom-left corners, light grey elsewhere, the curve of a shoe’s toe. On the right, the same 64 pixels as numbers. The black pixels are 0; the boot’s pixels are mostly between 170 and 234. The outlined pixel in row 4, column 1 is 98, where the toe starts.

What this tells us. The background is 0 and the boot is about 200. One image is 28 × 28 = 784 such numbers, and the model gets nothing else: no colour, no “this is a boot”.

PyTorch’s image layers expect a tensor of shape (images, channels, rows, columns). A channel is one layer of numbers for each pixel. A grey image has one channel; a colour photo has three, for red, green and blue. So one image is (1, 28, 28), and a batch of 64 is (64, 1, 28, 28). This is the batch dimension of Tensors and shapes, with two more dimensions for the picture.

Split and baseline

The test file stays sealed until the end, as in Train, validation and test sets. From the training file, Kwame takes a stratified sample of 10,000 images to train on and 5,000 others to validate on. Why not all 55,000? Each network then trains in about a minute on a CPU, and the ideas are the same. The cost is a few points of accuracy, which the limits section measures.

clothing.py

import torch
from sklearn.model_selection import train_test_split

rows = np.arange(len(labels))
train_rows, rest = train_test_split(rows, train_size=10_000, stratify=labels, random_state=0)
valid_rows, _ = train_test_split(rest, train_size=5_000, stratify=labels[rest], random_state=0)


def to_tensors(imgs, labs):
    """uint8 images (N, 28, 28) -> floats (N, 1, 28, 28) from 0 to 1, and int64 labels (N,)."""
    x = torch.tensor(imgs, dtype=torch.float32).unsqueeze(1) / 255
    return x, torch.tensor(labs, dtype=torch.long)


x_train, y_train = to_tensors(images[train_rows], labels[train_rows])
x_valid, y_valid = to_tensors(images[valid_rows], labels[valid_rows])
x_test, y_test = to_tensors(test_images, test_labels)
show_shape("x_train", x_train)
show_shape("x_valid", x_valid)
show_shape("x_test", x_test)
print("validation images per class:", torch.bincount(y_valid).tolist())

The output:

x_train      shape (10000, 1, 28, 28)
x_valid      shape (5000, 1, 28, 28)
x_test       shape (10000, 1, 28, 28)
validation images per class: [500, 500, 500, 500, 500, 500, 500, 500, 500, 500]

unsqueeze(1) adds the channel dimension at position 1, and dividing by 255 puts every pixel between 0 and 1. show_shape() is the notebook’s helper that prints a shape, the course’s habit.

Then the baselines. The majority class is right 10% of the time. Logistic regression, from the machine learning course, reads the 784 pixels as 784 features. flatten(1) keeps dimension 0 (the images) and lays out the rest in one row: (10000, 784).

clothing.py

from sklearn.linear_model import LogisticRegression

majority = int(torch.bincount(y_train).argmax())
print("majority class, validation accuracy:", round((y_valid == majority).float().mean().item(), 4))
logreg = LogisticRegression(max_iter=1000)
logreg.fit(x_train.flatten(1).numpy(), y_train.numpy())       # 784 pixel features per image
print("logistic regression, validation accuracy:", round(logreg.score(x_valid.flatten(1).numpy(), y_valid.numpy()), 4))

The output:

majority class, validation accuracy: 0.1
logistic regression, validation accuracy: 0.8346

What this tells us. A linear model on raw pixels gets 83.5% right, about 5 photos in 6. That is the number to beat. A network must do clearly better to be worth its cost.

Models

The training loop, with early stopping

fit() is the loop of Batches, epochs and validation: Adam with a learning rate of 0.001, shuffled batches of 64, the five steps, and an evaluation pass after each epoch. It adds early stopping from Overfitting controls. It copies the weights at each new best validation loss, stops after 5 epochs without a new best, and puts the best weights back. The notebook has the full function; its core is the familiar loop:

clothing.py

        for start in range(0, len(order), batch_size):
            batch = order[start:start + batch_size]
            images, labels = x_train[batch], y_train[batch]
            if flip:  # data augmentation: mirror about half of the batch
                mirror = torch.rand(len(batch)) < 0.5
                images = torch.where(mirror[:, None, None, None], images.flip(-1), images)
            optimizer.zero_grad()
            loss = loss_fn(model(images), labels)
            loss.backward()
            optimizer.step()

A multilayer network

First, the network of the earlier modules, with pixels in place of word counts. nn.Flatten() makes each image a row of 784 numbers. One hidden layer of 128 ReLU neurons follows, then 10 scores. A network of fully connected layers like this is often called a multilayer perceptron, or MLP.

clothing.py

class MLP(nn.Module):
    def __init__(self, hidden=128):
        super().__init__()
        self.layers = nn.Sequential(nn.Flatten(), nn.Linear(784, hidden), nn.ReLU(), nn.Linear(hidden, 10))

    def forward(self, x):
        return self.layers(x)


set_seed(0)
mlp = MLP()
print("weights:", count(mlp))
mlp_history, mlp_best = fit(mlp, x_train, y_train, x_valid, y_valid)
print("stopped after epoch", len(mlp_history), "| best:", mlp_best)

The output:

weights: 101770
stopped after epoch 24 | best: {'epoch': 19, 'train_loss': 0.2207, 'train_acc': 0.9249, 'valid_loss': 0.3723, 'valid_acc': 0.8712}

What this tells us. The MLP reaches 87.1% at epoch 19, 3.7 points above logistic regression. Early stopping waited 5 more epochs and gave the weights of epoch 19 back.

The problem: a flat network does not know where a pixel is

Look again at nn.Flatten(). It puts the 784 pixels in a row, and the first layer gives each position its own weights. Nothing tells the network that pixel 100 is next to pixel 101. Kwame tests this with a strange experiment: he puts the pixels of every image in one fixed random order, so that a pair of trousers looks like noise. Then he trains the same MLP again:

A training image, and the same 784 pixels in a fixed random order. The second picture means nothing to a person.

Text description of the chart

Left: a pair of trousers in grey on black. Right: the same pixel values scattered over the square, like television noise, with no shape left.

scrambled pixels, best: {'epoch': 19, 'train_loss': 0.225, 'train_acc': 0.9209, 'valid_loss': 0.3661, 'valid_acc': 0.8708}

87.1%, the same as before. For the MLP, a picture and its scrambled copy are the same problem, because it never used the picture’s layout. So it must learn “a collar here” separately for every position, from examples. The cost shows when a product is not exactly where the training photos had it. Kwame moves every validation image a few pixels to the right:

clothing.py

for pixels in (1, 2, 3):
    moved = torch.roll(x_valid, shifts=pixels, dims=3)   # the background is black, so this moves the item
    print(f"moved {pixels} pixel(s) right: MLP accuracy {evaluate(mlp, moved, y_valid)[1]:.4f}")

The output:

moved 1 pixel(s) right: MLP accuracy 0.7602
moved 2 pixel(s) right: MLP accuracy 0.4308
moved 3 pixel(s) right: MLP accuracy 0.2330

A shirt moved by two pixels is still a shirt to any person. The MLP falls from 87% to 43%. Its weights belong to positions, not to shapes.

The idea: a small pattern detector that slides

Think of searching a page for the word “refund” with a magnifying glass. You look at a small part, move the glass a little, and look again. You use the same eyes everywhere, and you find the word wherever it is.

A convolution does that with a filter: a small grid of weights, here 3 × 3. The filter sits on 3 × 3 pixels and multiplies each pixel by its weight. Then it adds the nine products, which gives one number. Then it moves one pixel and does it again, over the whole image. The result is a new 28 × 28 grid, the response map: high where the pattern is, low where it is not. The same nine weights are used at every position. So a pattern is found wherever it is, and nine weights cover the whole image. The name comes from mathematics, where this sliding sum is called a convolution. A network with such layers is a convolutional neural network (CNN), and it learns the weights of its filters as it learns any other weight.

Here is one step with a filter that Kwame made by hand: −1 on the left, +1 on the right. It responds to a left edge, where dark pixels meet bright pixels.

One step of a convolution on a real trouser image. The 3 × 3 window at row 8, column 11 (outlined) holds dark pixels on the left and bright ones on the right. Times the filter, it gives 2.78, the brightest point of the response map. Mid-grey in the map means 0; white is a left edge, black a right edge.

Text description of the chart

Left: a 28 by 28 image of trousers, with a small 3 by 3 square outlined at the left edge of one leg. Middle: the nine pixel values of that square: 0.00, 0.02, 0.98 in the first row; 0.00, 0.00, 0.92 in the second; 0.00, 0.47, 0.88 in the third. Times the filter, which is −1, 0, 1 in every row, they give 2.78: 0.98 + 0.92 + 0.88, because the left column is 0 and the middle column is multiplied by 0. Right: the response map, mid-grey where the image is flat, white along the left edges of the trouser legs and black along their right edges, with the matching pixel outlined.

Where the window is flat, all black or all bright, the left and right columns cancel, and the answer is 0. The notebook asks you to compute the 2.78 yourself.

The network uses two more ideas. Max pooling keeps the largest value of each 2 × 2 square. That halves the height and the width, so a shift of one pixel matters less. And a second convolution reads the first one’s response maps, so it can find patterns of patterns, such as a collar made of edges.

A small convolutional network

Kwame’s network has two convolution layers, with 16 and then 32 filters. Each is followed by ReLU and max pooling, and then come the familiar layers to 10 scores. He prints the shape after every layer, for a batch of 64, so that he is not guessing:

clothing.py

class SmallCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size=3, padding=1), nn.ReLU(), nn.MaxPool2d(2),   # 16 filters on 1 channel
            nn.Conv2d(16, 32, kernel_size=3, padding=1), nn.ReLU(), nn.MaxPool2d(2),  # 32 filters on 16 channels
        )
        self.classify = nn.Sequential(nn.Flatten(), nn.Linear(32 * 7 * 7, 64), nn.ReLU(), nn.Linear(64, 10))

    def forward(self, x):
        return self.classify(self.features(x))


set_seed(0)
cnn = SmallCNN()
x = x_train[:64]
show_shape("input", x)
for layer in cnn.features:
    x = layer(x)
    show_shape(type(layer).__name__, x)
show_shape("scores", cnn(x_train[:64]))
print("weights:", count(cnn), "| first convolution:", count(cnn.features[0]))

The output:

input        shape (64, 1, 28, 28)
Conv2d       shape (64, 16, 28, 28)
ReLU         shape (64, 16, 28, 28)
MaxPool2d    shape (64, 16, 14, 14)
Conv2d       shape (64, 32, 14, 14)
ReLU         shape (64, 32, 14, 14)
MaxPool2d    shape (64, 32, 7, 7)
scores       shape (64, 10)
weights: 105866 | first convolution: 160

What this tells us. nn.Conv2d(1, 16, kernel_size=3) makes 16 filters of 3 × 3 on 1 channel. Each filter gives one response map, so the channels go from 1 to 16. padding=1 adds a border of zeros, so the window fits at every pixel and the size stays 28 × 28. Pooling halves it to 14, the second block to 7, and 32 × 7 × 7 = 1,568 numbers go into the last layers. The first convolution has 16 × 9 weights + 16 biases = 160. The whole network has 105,866 weights, about as many as the MLP’s 101,770. Any difference comes from how the weights are used, not from how many there are.

Common mistake

The missing channel dimension

Symptom: Lena, a new developer on the team, feeds the network images of shape (64, 28, 28), straight from the file, without unsqueeze(1). “PyTorch will figure it out.” It stops with:

RuntimeError: Given groups=1, weight of size [16, 1, 3, 3], expected input[1, 64, 28, 28] to have 1 channels, but got 64 channels instead

Cause: nn.Conv2d accepts a 3-D input as one image without a batch: (channels, rows, columns). So it read the 64 images as one image with 64 channels, and its filters expect 1 channel. The error names the shape it built, [1, 64, 28, 28].

Fix: give the batch dimension and the channel dimension explicitly: x.unsqueeze(1) gives (64, 1, 28, 28). Print the shape before the first layer.

What it means for a real AI service: the upload page sends one photo at a time. One photo of shape (1, 28, 28), without the batch dimension, passes both convolutions and fails only in the last layers: mat1 and mat2 shapes cannot be multiplied (32x49 and 1568x64). That message points far from the cause. Make the shape explicit where images enter the service, as suggest() does below with reshape(1, 1, 28, 28).

Training curves

Kwame trains the CNN with the same fit(), from the same seed. It takes about 40 seconds on the test computer.

clothing.py

cnn_history, cnn_best = fit(cnn, x_train, y_train, x_valid, y_valid)
print("stopped after epoch", len(cnn_history), "| best:", cnn_best)

The output:

stopped after epoch 24 | best: {'epoch': 19, 'train_loss': 0.1723, 'train_acc': 0.9396, 'valid_loss': 0.3221, 'valid_acc': 0.8912}

• CNN (dashed: training)
• MLP (dashed: training)

Training loss (dashed) and validation loss (solid) per epoch, on 10,000 training and 5,000 validation images. Both networks stopped 5 epochs after their best epoch, 19.

Text description of the chart

Four lines over 24 epochs. All start near 0.6 to 0.66. The CNN’s validation loss falls to 0.40 at epoch 4, 0.35 at epoch 10 and its lowest value, 0.322, at epoch 19, then rises to between 0.33 and 0.38. The CNN’s training loss keeps falling, to 0.15 at epoch 24. The MLP’s validation loss stays above the CNN’s from epoch 4 on, with a lowest value of 0.372 at epoch 19; its training loss falls to about 0.21. Both validation curves are jagged: some epochs are worse than the one before.

What this tells us. The two curves of each network fall together at first and then separate. The training loss keeps falling while the validation loss stops, the overfitting picture of Module 3. The CNN’s validation loss is lower than the MLP’s at almost every epoch. The jagged epochs (8, 12, 17) are why early stopping waits 5 epochs instead of one.

And the moved images? The same test, with both networks:

moved 1 pixel(s) right: MLP 0.7602   CNN 0.8572
moved 2 pixel(s) right: MLP 0.4308   CNN 0.7346
moved 3 pixel(s) right: MLP 0.2330   CNN 0.4678

The CNN loses much less, but it is not immune: at three pixels it also fails on half of the images. Pooling forgives a small shift, but every training photo was centred, so the network never learned more. The numbers script also trained the CNN on the scrambled pixels. It drops to 85.4%, below the MLP, because its filters now look at neighbours that are not real neighbours. The layout is what it uses.

A fair comparison

Every model on the same 5,000 validation images, after the same split, with accuracy as the metric and the loss next to it:

              model  weights  valid_acc  valid_loss
     majority class        0     0.1000         NaN
logistic regression     7850     0.8346         NaN
                MLP   101770     0.8712      0.3723
          small CNN   105866     0.8912      0.3221

Validation accuracy of the four models on the same 5,000 images. The CNN is best, 2 points above the MLP and 5.7 above logistic regression.

Text description of the chart

A column chart with four columns. Majority class: 10%. Logistic regression: 83.5%. MLP: 87.1%. Small CNN: 89.1%.

What this tells us. Each step is a real gain. A hidden layer adds 3.7 points, and convolutions add 2.0 more with about the same number of weights. The CNN’s loss is also clearly lower (0.322 against 0.372). It costs more time, though: about 40 seconds against 5 for the MLP, because each filter works at 784 positions.

Where it fails

The confusion matrix of the machine learning course, now with ten classes: one row per true class, one column per predicted class. The diagonal holds the right answers.

Confusion matrix of the small CNN on the 5,000 validation images (500 per class). Darker cells hold a larger share of their row. Most errors fall in the block of tops: T-shirt/top, pullover, coat and shirt.

Text description of the chart

Ten rows and ten columns. Right answers on the diagonal, out of 500 each: T-shirt/top 421, trouser 493, pullover 435, dress 452, coat 418, sandal 474, shirt 330, sneaker 476, bag 477, ankle boot 480. The largest errors: shirts predicted as T-shirt/top 51, as pullover 50 and as coat 48; T-shirts/tops predicted as shirt 46; coats predicted as pullover 41; pullovers as coat 30; coats as shirt 28; pullovers as shirt 23; ankle boots as sneaker 19; sandals as sneaker 16; sneakers as ankle boot 15. Trousers, bags and the three shoe classes have few errors.

What this tells us. Two families cause most of the trouble. The tops: only 330 of 500 shirts are called “Shirt”, a recall of 0.66. Shirts become T-shirts (51), pullovers (50) and coats (48), and 46 T-shirts become shirts. The shoes: ankle boots, sneakers and sandals are sometimes swapped. Trousers and bags are almost always right. One accuracy number, 89.1%, hides a category where one shirt in three is misfiled.

Kwame looks at the errors themselves: first the five that the network is most sure of, then its five surest errors on shirts.

Real errors of the CNN on validation images. The first five are its most confident errors overall; the last five are its most confident errors on images labelled Shirt. The probability is the one the network gave its wrong answer.

Text description of the chart

Ten images, each outlined, with the label, the model’s answer and its probability. The first five: an ankle boot called sneaker (1.00), a T-shirt/top called dress (1.00), a sneaker called ankle boot (1.00), another ankle boot called sneaker (1.00), and a sandal called sneaker (1.00). The last five, all labelled Shirt: one dark, narrow garment called trouser (1.00), one called coat (0.99), and three called pullover (0.99, 0.99, 0.98). Several look like what the model said: a long, narrow top that could be a dress, shirts with long sleeves and no visible collar that look like pullovers, and a low ankle boot that looks like a high sneaker.

What this tells us. Look at the pictures before you blame the model. At 28 × 28 pixels, a shirt with long sleeves and no visible collar looks like a pullover. A low ankle boot looks like a sneaker. Some labels look doubtful, such as the dark, narrow “shirt” that the network calls trousers. The network is sure of these answers (0.98 to 1.00). So a high probability does not prove a right answer. A few of these would fool a person too. For Ines, this means suggestions for tops need the most care.

Data augmentation: a flip

A shirt flipped from left to right is still a shirt. So a flipped copy is a free extra training image with a true label, as in Overfitting controls. fit() already has the lines: torch.rand(len(batch)) < 0.5 chooses about half of each batch, and images.flip(-1) mirrors them along the last dimension, the columns. Kwame trains the same network with the flip, from the same seed:

clothing.py

set_seed(0)
cnn_flip = SmallCNN()
flip_history, flip_best = fit(cnn_flip, x_train, y_train, x_valid, y_valid, flip=True)
print("without flip:", cnn_best)
print("with flip:   ", flip_best)

The output:

without flip: {'epoch': 19, 'train_loss': 0.1723, 'train_acc': 0.9396, 'valid_loss': 0.3221, 'valid_acc': 0.8912}
with flip:    {'epoch': 16, 'train_loss': 0.2482, 'train_acc': 0.9103, 'valid_loss': 0.3206, 'valid_acc': 0.8854}

The loss is a little lower and the accuracy a little worse. One run cannot decide, because a different seed gives a different result. So the numbers script trained both versions with five seeds. Kwame’s rule: keep the flip only if it beats the plain network by more than the spread between seeds.

Seed	0	1	2	3	4	Mean
Without flip, validation accuracy	89.1%	88.9%	89.2%	87.7%	88.7%	88.7%
With flip, validation accuracy	88.5%	89.0%	89.4%	88.6%	88.5%	88.8%
Difference (points)	−0.6	0.0	+0.1	+0.8	−0.2	+0.04

What this tells us. The flip makes no measurable difference here: +0.04 points on average, while single seeds move from −0.6 to +0.8. The mean validation loss is a little lower with the flip (0.315 against 0.322), on three seeds of five. That is too small to matter for Ines.

Why so little? The data gives a reason. A short script counts which half of each image is brighter. In the training file, all 6,000 ankle boots and 99.4% of the sneakers point their toe to the left. The flip makes boots that point right, which never appear in this catalogue or its test set. The label stays true, but the new images are unlike the photos that the model will see. On real seller photos, where shoes point both ways, the flip could matter much more.

By the rule, the final model is the plain CNN: the simpler training, and no worse.

The operating point

Ines does not want a category for every photo. She wants it filled in when the model is sure, and a question to the seller otherwise. The model’s top probability is the measure of “sure”. On the validation images, Kwame checks four thresholds:

clothing.py

top = valid_probs.max(1).values
for threshold in (0.5, 0.8, 0.9, 0.95):
    sure = top >= threshold
    in_top2 = (valid_probs[~sure].topk(2, dim=1).indices == y_valid[~sure, None]).any(1)
    print(f"threshold {threshold}: filled in {sure.float().mean():.3f}, right there {(valid_pred[sure] == y_valid[sure]).float().mean():.4f}, "
          f"others: true class in the top 2 {in_top2.float().mean():.4f}")

The output:

threshold 0.5: filled in 0.976, right there 0.9035, others: true class in the top 2 0.7227
threshold 0.8: filled in 0.819, right there 0.9556, others: true class in the top 2 0.8826
threshold 0.9: filled in 0.735, right there 0.9763, others: true class in the top 2 0.9011
threshold 0.95: filled in 0.662, right there 0.9861, others: true class in the top 2 0.9201

What this tells us. A higher threshold fills in fewer photos and gets more of them right. Ines and Kwame choose 0.9. About 3 photos in 4 (73.5%) get their category filled in, and 97.6% of those are right. For the other quarter, the seller sees the two most likely categories, and the right one is among them 90% of the time.

The final test, once

Every choice is made: the plain CNN, its best epoch, the threshold 0.9. Kwame opens the 10,000 test images once:

clothing.py

test_probs = predict(cnn, x_test)
test_pred = test_probs.argmax(1)
sure = test_probs.max(1).values >= 0.9
in_top2 = (test_probs[~sure].topk(2, dim=1).indices == y_test[~sure, None]).any(1)
print("test accuracy:", round((test_pred == y_test).float().mean().item(), 4))
print(f"filled in {sure.float().mean():.3f}, right there {(test_pred[sure] == y_test[sure]).float().mean():.4f}, "
      f"others: true class in the top 2 {in_top2.float().mean():.4f}")
test_cm = confusion_matrix(y_test, test_pred)
print("recall per class:", dict(zip(CLASSES, (test_cm.diagonal() / test_cm.sum(1)).round(3).tolist())))

The output:

test accuracy: 0.8879
filled in 0.725, right there 0.9742, others: true class in the top 2 0.9038
recall per class: {'T-shirt/top': 0.827, 'Trouser': 0.981, 'Pullover': 0.842, 'Dress': 0.893, 'Coat': 0.853, 'Sandal': 0.966, 'Shirt': 0.661, 'Sneaker': 0.95, 'Bag': 0.936, 'Ankle boot': 0.97}

What this tells us. On 10,000 photos that no step has touched, the CNN gets 88.8% right, a little below its 89.1% on validation, as expected. At the threshold, it fills in 72.5% of photos, and 97.4% of those are right. Shirts are still the weak class (recall 0.661). These are the numbers to report, and nothing changes after reading them.

Save the model and the experiment record

Brightloom’s upload page will load the model in another program, maybe months later. So Kwame saves two things. The first is the weights, with state_dict(): a dictionary of the network’s tensors, by layer name. It is not the whole Python object, so any code with the same SmallCNN class can load it. The second is an experiment record: everything someone needs to repeat and judge the result. Save and inspect results treats both in detail.

clothing.py

torch.save(cnn.state_dict(), "fashion_cnn.pt")
record = {
    "model": "SmallCNN: conv 16 (3x3) - pool - conv 32 (3x3) - pool - linear 64 - linear 10",
    "weights": count(cnn),
    "data": {"name": "Fashion-MNIST (Zalando Research, MIT License)", "sha256": FILES,
             "train": "10,000 of the training file, stratified, random_state=0",
             "valid": "5,000 others of the training file, stratified, random_state=0", "test": "the 10,000 test images"},
    "training": {"optimizer": "Adam", "lr": 0.001, "batch_size": 64, "max_epochs": 30, "patience": 5,
                 "seed": 0, "flip": False, "best_epoch": cnn_best["epoch"], "torch_threads": torch.get_num_threads()},
    "results": {"valid_acc": cnn_best["valid_acc"], "valid_loss": cnn_best["valid_loss"],
                "test_acc": round((test_pred == y_test).float().mean().item(), 4),
                "fill_in_threshold": 0.9, "test_share_filled_in": round(sure.float().mean().item(), 4)},
    "versions": {"python": platform.python_version(), "torch": torch.__version__, "numpy": np.__version__,
                 "scikit-learn": sklearn.__version__},
    "limits": "28 x 28 grey catalogue photos, one centred product on black; not tested on real seller photos.",
}
Path("experiment.json").write_text(json.dumps(record, indent=2))
print(sorted(p.name for p in Path(".").iterdir() if not p.name.startswith(".")))
print("fashion_cnn.pt:", Path("fashion_cnn.pt").stat().st_size, "bytes")

The output:

['data', 'experiment.json', 'fashion_cnn.pt']
fashion_cnn.pt: 427333 bytes

Then he loads the weights into a new network, as the upload page would, and compares its answers:

clothing.py

loaded = SmallCNN()
loaded.load_state_dict(torch.load("fashion_cnn.pt", weights_only=True))
loaded.eval()                                   # evaluation mode before any use
same = torch.equal(predict(loaded, x_test).argmax(1), test_pred)
print("same predictions on all 10,000 test images:", same)

The output:

same predictions on all 10,000 test images: True

What this tells us. The file is 427 KB: 105,866 weights of 4 bytes each, plus the layer names. weights_only=True loads only tensors, so a file from someone else cannot run code when you load it. And loaded.eval() sits in the loading code, not in a person’s memory, as Module 4 advised.

A prediction on new images

suggest() is what the upload page would call: one photo in, and either a category to fill in or two to ask about. Kwame gives it the first eight test images. The model never trained on them.

clothing.py

def suggest(model, image, threshold=0.9):
    """One 28 x 28 image (0..255) -> what the upload page shows, and the top 3 probabilities."""
    x = torch.tensor(np.asarray(image), dtype=torch.float32).reshape(1, 1, 28, 28) / 255
    probs = predict(model, x)[0]
    top = probs.topk(3)
    choices = [(CLASSES[c], round(p, 3)) for p, c in zip(top.values.tolist(), top.indices.tolist())]
    shown = f"fill in: {choices[0][0]}" if choices[0][1] >= threshold else f"ask: {choices[0][0]} or {choices[1][0]}?"
    return shown, choices


for i in range(8):
    shown, choices = suggest(loaded, test_images[i])
    print(f"{i}  label {CLASSES[test_labels[i]]:<11} -> {shown:<28} {choices}")

The output:

0  label Ankle boot  -> fill in: Ankle boot          [('Ankle boot', 0.976), ('Sneaker', 0.016), ('Sandal', 0.007)]
1  label Pullover    -> fill in: Pullover            [('Pullover', 0.938), ('Coat', 0.06), ('Shirt', 0.001)]
2  label Trouser     -> fill in: Trouser             [('Trouser', 1.0), ('Dress', 0.0), ('Sandal', 0.0)]
3  label Trouser     -> fill in: Trouser             [('Trouser', 1.0), ('Dress', 0.0), ('Sandal', 0.0)]
4  label Shirt       -> ask: Shirt or Coat?          [('Shirt', 0.867), ('Coat', 0.056), ('T-shirt/top', 0.031)]
5  label Trouser     -> fill in: Trouser             [('Trouser', 1.0), ('Dress', 0.0), ('Pullover', 0.0)]
6  label Coat        -> fill in: Coat                [('Coat', 0.994), ('Pullover', 0.005), ('Shirt', 0.001)]
7  label Shirt       -> ask: Coat or Shirt?          [('Coat', 0.59), ('Shirt', 0.391), ('Pullover', 0.014)]

The first eight test images with the saved model's two most likely categories and the true label. The outlined image is the one the model would have got wrong; at the threshold of 0.9, the page asks the seller instead.

Text description of the chart

Eight images with the model’s top two answers. An ankle boot: ankle boot 0.98, sneaker 0.02. A pullover: pullover 0.94, coat 0.06. Three trousers: trouser 1.00 each. A shirt: shirt 0.87, coat 0.06. A coat: coat 0.99, pullover 0.01. A second shirt, outlined: coat 0.59, shirt 0.39.

What this tells us. Six photos get their category filled in, and all six are right. The two shirts stay below 0.9, so the page asks. For photo 4 it asks “Shirt or Coat?”, and the model’s first guess is right. For photo 7, the first guess, coat, is wrong, but “shirt” is the second choice, so the seller can still pick it with one click. This is the operating point at work: the model’s doubt goes to a person instead of the wrong aisle.

Recommendations

Kwame writes to Ines in her words:

• Use the model to suggest categories on the upload page. Fill in the category when the model is at least 90% sure: about 7 photos in 10, with about 97 in 100 right. Otherwise, show the two most likely categories; the right one is among them about 9 times in 10.
• Expect the most trouble with tops. One shirt in three is misread as a T-shirt, pullover or coat. Shirts will often get the question instead of a filled-in category, which is the safe outcome.
• Before launch, test it on Brightloom’s own photos. Every number here comes from small, grey, centred catalogue photos. Label a few hundred real seller photos and measure again before anyone relies on these numbers.
• Keep the seller in charge. The page must make the suggestion easy to change, and the team should count how often sellers change it.

And what not to use it for: to reject or hide a listing, to judge whether a photo is genuine, or on photos of people.

Limits and next steps

• Tiny, clean photos. Fashion-MNIST photos are 28 × 28, grey, centred, on black, with one product each. Seller photos are large, in colour, and taken on beds, floors and hangers. This model will not work on them without new data. The shift test shows how fragile it is, even to a move of 3 pixels.
• A sample of the data. The networks trained on 10,000 images to keep the notebook short. The numbers script trained the same CNN once on all 55,000 non-validation images. It reached 92.0% validation accuracy at epoch 9, about three points more, in about 2 minutes. More data helped more than any change of architecture here.
• One catalogue. The products come from one European online shop around 2017. Styles, cultures and price ranges outside it are under-represented, and every shoe faces the same way.
• One look at the test set too early. While we prepared this case study, an earlier version of the numbers script printed the test accuracy of the flip model (87.7%) before the rule above was fixed. The rule rests only on the five validation seeds, and the test result on this page is the only one for the model that the rule chose. Still, a strict process keeps even one early look out, so treat the test number as a little less independent than it should be. In your own work, write the rule down and commit it before any test number exists.
• Labels. Some labels look wrong, and some photos are unclear even to a person. That puts a ceiling on accuracy, especially for tops.
• Next steps. For real photos, start from a network that was pretrained on millions of images, and fine-tune it on Brightloom’s labelled photos. The case study Fine-tune a small transformer measures what pretraining buys for text; the same idea works for images. Then monitor how often sellers change the suggestion, per category.

Responsible use

This model sorts product photos; it judges nobody. Keep it that way:

• A suggestion, not a decision. The seller confirms or changes every category. Do not hide, demote or reject a listing because of the model.
• Products only. Do not use it on photos of people. It was never trained or tested for that. A classifier of clothes on bodies also raises privacy and fairness questions that this case study does not answer.
• Unequal errors. Errors are not spread evenly: shirts fail far more often than trousers. If some sellers mainly sell shirts, their listings get more questions and more wrong suggestions. Report the error rate per category, and watch it after launch.
• Honest claims. Say “88.8% on Fashion-MNIST test images”, never “88.8% on Brightloom”. The second has not been measured.

Make it a portfolio piece

This case study shows an employer that you can take image data from raw bytes to a tested, saved model and an honest decision. Make it yours with one change. For example, use all 55,000 training images, add a third convolution layer, or try a shift of one or two pixels as augmentation. Then make it public, easy to read in five minutes.

What to put in the repository:

• README.md, the page that people read first (template below).
• The notebook, run from top to bottom, with its outputs.
• experiment.json from the notebook, and the confusion matrix as a picture.
• fashion_cnn.pt (427 KB) with the SmallCNN class and the PyTorch version that loads it, or only the code, because a reader can retrain it in a minute.
• A short model card, as in Document use boundaries in the machine learning course: intended use, not for, data, evaluation, limits.
• The data source and licence: Fashion-MNIST, Zalando Research, MIT License. A link and the checksums are enough; do not copy the 31 MB.

A README template:

# Suggesting product categories from photos

## Problem
An online marketplace wants to suggest one of ten clothing categories from the
product photo: fill it in when the model is sure, ask the seller otherwise.

## Data and licence
Fashion-MNIST (Zalando Research, 2017), 70,000 grey 28x28 catalogue photos,
10 classes, MIT License, github.com/zalandoresearch/fashion-mnist. Trained on a
stratified sample of 10,000; validated on 5,000; tested once on the 10,000 test images.

## Approach
Baselines (majority 10%, logistic regression 83.5%), an MLP (87.1%) and a small
CNN (89.1%), all with the same split; early stopping on validation loss. A
left-right flip tested over five seeds: no measurable gain, so not used.

## Results (test set, used once)
Accuracy 88.8%. At a 0.9 threshold: 72.5% of photos filled in, 97.4% of them right;
for the others, the true class is in the top two 90% of the time. Weakest class:
shirt (recall 0.66).

## Limits
Tiny, grey, centred catalogue photos; not tested on real seller photos.

How to describe it in an interview: “I built a small convolutional network that suggests a clothing category from a product photo. It beat logistic regression and a plain network under the same split, and gave 88.8% on a held-out test. When it fills the category in by itself, it is right 97% of the time. I also showed why the plain network fails when an item moves and found that shirts cause most errors. A flip augmentation did not help on this data, and I measured why.”

Go further:

• Train on all 55,000 images, and plot validation accuracy against the number of training images (5,000, 10,000, 20,000, 55,000).
• Try a shift of one or two pixels as augmentation. Predict first: will it help the moved-image test more than the plain accuracy?
• Look at the 16 filters of the first convolution layer as 3 × 3 pictures, and at their response maps for one shirt.
• Measure the time of one prediction, as in Inference costs, and compare it with the MLP.

Knowledge check

Check your understanding

The MLP reached the same validation accuracy (87.1%) on images whose pixels were put in a fixed random order. What does that show, and how does a convolution change it?

Show the answer

It shows that the MLP never used the layout of the image. nn.Flatten() gives every pixel position its own weights, and nothing tells the network which pixels are neighbours. So a scrambled image is the same problem to it, and it must learn each pattern separately at every position. That is why moving the images by two pixels dropped it from 87% to 43%. A convolution slides one small filter over the image and multiplies neighbouring pixels by the same weights everywhere. So it uses the layout and finds a pattern wherever it is, and with pooling it forgives small shifts (73% at two pixels). On scrambled pixels, the CNN lost that advantage and fell to 85.4%.

Check your understanding

The flip gave +0.8 points on one seed and -0.6 on another. Kwame decided not to use it. Was that right, and why might the flip help more on real seller photos?

Show the answer

Yes. Over five seeds, the mean gain was +0.04 points. Single seeds moved between −0.6 and +0.8. A gain smaller than the spread between seeds is not a measured gain. So the simpler training wins. The data also explains the result. Every ankle boot and almost every sneaker in Fashion-MNIST points its toe to the left. The flip made shoes that point right, which never appear in this catalogue’s test photos. The label stayed true, but the images were unlike the real ones. Seller photos show shoes both ways. There, the flip adds realistic examples and could help.

Check your understanding

Ines asks why the page does not simply fill in the most likely category for every photo. Use the numbers to answer.

Show the answer

Then about 11 photos in 100 would get a wrong category, and many sellers would not notice. Those products would sit in the wrong aisle. With a threshold of 0.9, about 7 photos in 10 are filled in, and 97.4% of those are right on the test set. For the other 3 in 10, the page asks the seller to choose between the two most likely categories. The right one is among them about 9 times in 10. So the model works alone where it is reliable. Where it is not, it passes its doubt to a person. The threshold is a business choice: a higher one fills in fewer photos and makes fewer silent errors.

Recap

• You read raw image files into a tensor of shape (images, 1, 28, 28) and checked the data before any model: balanced classes, no repeats.
• Under one split and one metric: majority class 10%, logistic regression 83.5%, an MLP 87.1%, and a small CNN 89.1%. The MLP and the CNN have about the same number of weights.
• A convolution is a small pattern detector that slides over the image. The MLP ignored the layout (the same accuracy on scrambled pixels) and failed when items moved; the CNN uses the layout.
• The confusion matrix and the real errors showed where it fails: shirts against T-shirts, pullovers and coats, and some unclear labels.
• A flip kept every label true but gave no measurable gain over five seeds, because every shoe in this catalogue faces the same way. The plain CNN stayed.
• The test, used once: 88.8%. At a 0.9 threshold, 72.5% of photos are filled in, 97.4% of them right. The model is saved as a state_dict with an experiment record, reloads with the same answers, and suggests categories for new photos.

Next, a different kind of data: a table of facts. There, a neural network must prove that it is worth its cost against gradient boosting.

Next: Deep network or gradient boosting?