Skip to content
Français

Fast enough on a CPU

  • Lesson 28
  • 40 min
  • Module 8: Run vision in production

Where we are

We can now see when the photos change under a model. The model must also answer quickly on the warehouse's ordinary computer, with no graphics card. In this lesson, we make the image smaller, batch the photos, export to ONNX and use 8-bit numbers, and we measure the time and the accuracy after each step.

What you will be able to do

Make a vision model fast enough for the computer that will run it, without losing accuracy by accident. Each change (a smaller image, a batch, a new file format, smaller numbers) can change both the time and the accuracy. So you measure both after every change, on the computer that matters.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It measures ResNet-18 at three image sizes, with a batch, exported to ONNX and quantised to int8. Its setup cells install PyTorch, ONNX and ONNX Runtime and download the catalogue photos and the ResNet-18 weights (47 MB). About 40 minutes, of which about 1 minute of computing on a laptop.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer (for the project steps; the notebook needs none of it)

  • Your larkvision project at the end of Module 8 (snapshot C42/snapshots/end-of-m08), set up in Domain shift, with its .venv environment.

All times in this lesson were measured on one laptop (Apple M4). Your times will differ. Tested on macOS with Python 3.12.13, PyTorch 2.15.0, ONNX 1.23.2 and ONNX Runtime 1.31.1. The Windows commands are written but were not run.

A worker at the returns desk picks up a candle, takes a photo and waits for the answer. If the answer takes a second, she waits a second. With 600 returns a day, the wait adds up, and the queue of boxes grows.

The time from “photo in” to “answer out” is the latency. For Hamid’s desk, the latency of one photo matters, because a person waits for each answer.

The problem: “small” and “fast” are not the same thing

Section titled “The problem: “small” and “fast” are not the same thing”

Larkfield’s returns desk has an ordinary office computer: no graphics card, and other programs running. Amira wants the inspector to use one thread (one line of work on the processor), so it does not slow down the rest.

Tomás has an easy answer: “Take the smallest model.” MobileNetV3 is a network made for phones: 22 MB against 47 MB for ResNet-18. Priya measured both on one thread, one photo at a time:

Model, in PyTorch, one thread Size Time a photo
ResNet-18 47 MB 10.2 ms
MobileNetV3 22 MB 84.9 ms

The smaller model was eight times slower on this laptop. MobileNetV3 uses kinds of layers that PyTorch runs badly on this processor. A model card cannot tell you this; only a measurement on your computer can.

The idea: one change at a time, two measures each time

Section titled “The idea: one change at a time, two measures each time”

There are five usual ways to make a vision model faster on a processor (CPU):

  1. A smaller image. A photo of 160 × 160 pixels has about half the pixels of 224 × 224. So the network does about half the work.
  2. A batch. Send several photos in one call. The overhead of a call is shared.
  3. Export to ONNX. ONNX (Open Neural Network Exchange) is a file format for a trained network. ONNX Runtime, a small library, runs the file without PyTorch.
  4. int8 quantisation. Store the weights, and compute, with 8-bit whole numbers instead of 32-bit decimal numbers (fp32).
  5. A smaller backbone. Another network, such as MobileNetV3.

After each change, measure two things: the time per photo, and the accuracy on the same test photos. Here, the accuracy is the catalogue test with a balanced logistic regression on the network’s embeddings, as in Module 3.

Three boxes from left to right. A PyTorch model, 47 MB, needs PyTorch. An arrow labelled export (one call) leads to an ONNX file in fp32, 45 MB, needs ONNX Runtime. An arrow labelled quantise (64 photos) leads to an ONNX file in int8, 11 MB, needs ONNX Runtime. Below: check the accuracy and the time after every step: each one can change both.
From PyTorch to a file that ONNX Runtime runs: export, then quantise. Sizes of ResNet-18 in this course.
Text description of the diagram

Three boxes in a row (stacked on a phone). The first is “PyTorch model”: 47 MB, needs PyTorch. An arrow, “export (one call)”, leads to “ONNX file (fp32)”: 45 MB, needs ONNX Runtime. A second arrow, “quantise (64 photos)”, leads to “ONNX file (int8)”, in green: 11 MB, needs ONNX Runtime. A sentence below says: check the accuracy and the time after every step, because each one can change both.

Four weights of a layer: −0.80, −0.10, 0.25 and 1.20. int8 stores whole numbers from −128 to 127, and one scale for the layer. The largest weight, 1.20, must become 127. So the scale is 1.20 ÷ 127 = 0.00945.

Weight (fp32, 4 bytes) ÷ scale Stored (int8, 1 byte) × scale again Error
−0.80 −84.7 −85 −0.803 0.003
−0.10 −10.6 −11 −0.104 0.004
0.25 26.5 26 0.246 0.004
1.20 127.0 127 1.200 0.000

Each weight takes 1 byte instead of 4, so the file is about four times smaller. Each weight moves by less than half a scale step. Whole-number arithmetic is also faster on most processors. The question is whether thousands of such small errors change the answers. Only a measurement tells.

For the numbers that flow through the network (not the weights), ONNX Runtime needs their range. It runs a few photos through the network to find it: this is calibration. The course used 64 training photos.

Larkfield: ResNet-18, one change at a time

Section titled “Larkfield: ResNet-18, one change at a time”

Priya measured every step on the course’s laptop, with one thread and one photo at a time:

Milliseconds per photo for ResNet-18 on the course's laptop (Apple M4), one thread, one photo at a time, after one warm-up call. PT is PyTorch, with the image size in pixels. fp32 and int8 are the ONNX files, at 224 pixels. Your computer will give other times.
Text description of the chart

A column chart with five columns. PyTorch at 224 pixels: 10.17 ms. PyTorch at 160 pixels: 6.22 ms. PyTorch at 128 pixels: 5.16 ms. ONNX at fp32, 224 pixels: 17.14 ms, the slowest. ONNX at int8, 224 pixels: 8.4 ms.

The whole table, with the accuracy and the size:

Change Accuracy Time a photo (1 thread) Batch of 16 (per photo) File
ResNet-18, PyTorch, 224 pixels 0.850 10.2 ms 16.7 ms 47 MB
Image 160 pixels 0.838 6.2 ms 8.9 ms 47 MB
Image 128 pixels 0.838 5.2 ms 7.0 ms 47 MB
ONNX fp32, 224 0.850 17.1 ms 23.4 ms 44.7 MB
ONNX int8, 224 0.856 8.4 ms 8.4 ms 11.3 MB
MobileNetV3, PyTorch 0.881 84.9 ms 59.8 ms 22 MB
MobileNetV3, ONNX fp32 0.881 7.1 ms 10.0 ms 16.8 MB
MobileNetV3, ONNX int8 0.869 3.4 ms 3.6 ms 4.7 MB

Read it one row at a time:

  • Smaller images saved time and cost 2 photos of 160 (0.850 → 0.838). One photo is 0.6 points, so this is close to noise.
  • A batch of 16 was slower per photo on one thread. Batching helps when many photos wait at once and the computer has threads to spare. Hamid’s desk sends one photo at a time.
  • ONNX fp32 was slower than PyTorch on one thread. Export is not a speed-up by itself. Its value is that the service no longer needs PyTorch.
  • int8 gave the speed back and made the file four times smaller. The accuracy did not fall here: 0.856 is one photo more than 0.850, which is noise.
  • MobileNetV3 is fast only after export: 84.9 ms in PyTorch, 3.4 ms in ONNX int8. The same network, two runtimes, 25 times apart.

The surprise of the batch, in the notebook

Section titled “The surprise of the batch, in the notebook”

The notebook repeats the ResNet-18 steps on your computer. On the course’s laptop, the solution notebook printed:

accuracy, PyTorch, 224 pixels: 0.85
ms a photo, one at a time, 1 thread: 9.8
ms a photo, batch of 16, 1 thread: 16.7
ms a photo, one at a time, 4 threads: 6.6
ONNX file: 44.7 MB
accuracy, ONNX fp32: 0.85
ms a photo, ONNX fp32, 1 thread: 16.9
int8 file: 11.3 MB
accuracy, ONNX int8: 0.856
ms a photo, ONNX int8, 1 thread: 8.7

A second run gave 9.4 ms and 8.4 ms for the same lines: times move by a few percent from run to run. Compare times that you measured on the same computer, in the same run. The accuracies do not move.

The image service does not classify products. It inspects returns with the small PatchCore of Module 6. Amira applied the same two steps to it: export the first layers of ResNet-18 to ONNX, then quantise to int8.

Small PatchCore Model file Memory of good patches Time a photo Test AUROC Cost a day (test)
32-bit (Module 6) 11.1 MB 15.4 MB 17.8 ms 0.847 1,290.8
int8 (the service) 2.9 MB 15.4 MB 15.1 ms 0.836 988.1

These times used four threads. Module 6’s table gave 78 MB for the same method, because it counted the whole ResNet-18 file and a memory in 32-bit numbers. Here, the model file holds only the layers the inspector uses, and the memory uses 16-bit numbers. int8 changed the scores a little, so Priya chose the lines again on the validation photos. The AUROC fell by 0.011. The cost per day fell too. But with 18 damaged test photos, that is not a real gain: one damaged candle more or less moves the cost by hundreds of dollars. The fair reading is “about the same quality, a quarter of the file size, and no PyTorch”.

The service needs only ONNX Runtime, NumPy, Pillow and OpenCV. The project’s environment grows from about 230 MB to 447 MB with them. This lesson’s notebook environment, with PyTorch, took about 1.2 GB.

class Inspector:
"""Loads models/inspector/: model.onnx, bank.npy and config.json (lines and validation statistics)."""
def __init__(self, folder, threads=1):
import onnxruntime as ort
folder = Path(folder)
self.config = json.loads((folder / "config.json").read_text())
so = ort.SessionOptions()
so.intra_op_num_threads = threads
self.session = ort.InferenceSession(str(folder / "model.onnx"), so, providers=["CPUExecutionProvider"])
self.bank = np.load(folder / "bank.npy").astype(np.float32)
self.bank_sq = (self.bank ** 2).sum(1)
self.version = self.config["version"]
def score(self, img):
f = self.session.run(None, {"image": preprocess(img)})[0][0]
q = f.reshape(f.shape[0], -1).T
d2 = (q ** 2).sum(1)[:, None] + self.bank_sq[None] - 2 * q @ self.bank.T
return float(np.sqrt(max(d2.min(1).max(), 0.0)))

intra_op_num_threads = threads fixes the number of threads, one by default. The score is the distance of the worst patch to its nearest good patch, as in Module 6.

  1. Open a terminal in your larkvision folder and activate the environment (source .venv/bin/activate, or .venv\Scripts\Activate.ps1 on Windows).

  2. In a new terminal, set LARKVISION_DATA and LARKVISION_MODELS again, with the commands of step 6 in Domain shift.

  3. Judge three test photos: a good candle, a candle with extra wax and a candle with a bent wick. On macOS and Linux:

    Terminal window
    vision inspect ../nextia-learning-labs/C42/data/returns/test/R-6a26ad64.jpg ../nextia-learning-labs/C42/data/returns/test/R-33640732.jpg ../nextia-learning-labs/C42/data/returns/test/R-1cc8cc10.jpg

    On Windows, write the paths with ..\nextia-learning-labs\C42\data\returns\test\.

    R-6a26ad64: pass score 5.832 confidence 0.33 37 ms
    R-33640732: check score 9.924 confidence 0.00 21 ms -> review: unsure: the score is between the two lines
    R-1cc8cc10: fail score 19.445 confidence 0.80 22 ms
  4. Read the times. The first photo takes longer (37 ms): the first call prepares the model. The next ones take about 21 ms. This time also includes a face finder, which the last lesson of this module explains.

The scores and zones should be the same on your computer, or very close. The times will not be. The next lesson explains the confidence and the review folder.

When to use each change, and what it costs

Section titled “When to use each change, and what it costs”

Use ONNX and int8 when the model must run on a CPU without PyTorch, as in Amira’s service. Use a smaller image when the details you need survive it. Remember the colour spot of When not to build it: under one pixel at 224 wide, it would vanish at 128. Use a batch when many photos wait at once, for example a night job over the whole catalogue.

Do not trust a speed-up you did not measure on the target computer. And do not skip the accuracy after int8: here it held, but on another network or other photos it can fall by several points.

What it costs: each change is a new model version. Its accuracy, time and file are measured again and written down. Export can fail on unusual layers, and calibration needs photos that look like the real ones. One more cost is not technical. The ResNet-18 and MobileNetV3 weights learned from ImageNet, so check their use in a product as in The licence is part of “ready”. Serving a model with versions and a way back is the subject of Deploying AI Models for Real Users.

Guided practice: the steps in the notebook

Section titled “Guided practice: the steps in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The packages cell prints one line per package. The data cell ends with a Ready: line.
  3. In section 1, run the cell. You see accuracy, PyTorch, 224 pixels: 0.85 and your time.
  4. In section 2, predict, then run the cell. You see 0.838.
  5. In section 3, run the cell. Compare the batch of 16 with one photo at a time on your computer.
  6. In section 4, run the export cell. You see accuracy, ONNX fp32: 0.85.
  7. In section 5, run the int8 cell. ONNX Runtime prints a warning about “pre-processing”: it is harmless here. You see int8 file: 11.3 MB and accuracy, ONNX int8: 0.856.

Change one thing: the image size, to 128 × 128 pixels, in PyTorch. Before you run it, predict: will the accuracy be closer to 0.838 or to 0.80? In the notebook, section 6, put the accuracy in acc_128, then run the check cell.

Show the answer
acc_128 = accuracy(pytorch, 128)
print(acc_128, ms_per_photo(pytorch, 128))
0.838 5.2

The same accuracy as at 160 pixels, and about half the time of 224. For product types, the shape and colour of the whole product survive at 128. For Hamid’s inspector, they would not: a small colour spot needs every pixel.

Common mistake

One timing, on the wrong computer

Symptom: Tomás times the inspector once on his own laptop, with all its threads, and writes “4 ms a photo” in the plan. On the desk computer, with one thread and other programs open, the first photos take ten times longer.

Cause: he measured one call, which includes the warm-up, on a different computer, with a different number of threads. The first call of vision inspect took 37 ms here, and the next ones about 21 ms.

Fix: measure on the target computer, or on one like it. Fix the number of threads. Make one warm-up call, then time many photos and report the middle value (p50) and a slow one (p95), as vision replay does.

Consequence in AI work: a latency promise that was never measured on the real computer breaks on the first busy day. Then the queue at the desk grows.

Check your understanding

A layer's weights are −0.60, 0.15 and 0.90. With int8 and one scale for the layer (the largest weight becomes 127), what is the scale, what is stored for 0.15, and what value comes back?

Show the answer

The scale is 0.90 ÷ 127 = 0.00709. For 0.15: 0.15 ÷ 0.00709 = 21.2, stored as 21. Back: 21 × 0.00709 = 0.149, an error of about 0.001. The weight takes 1 byte instead of 4.

Check your understanding

On one thread, the exported ONNX fp32 ResNet-18 was slower than PyTorch (17.1 against 10.2 ms). Why did Amira still export it?

Show the answer

Export is not mainly a speed-up. The ONNX file runs with ONNX Runtime alone, so the service does not need PyTorch. That makes it smaller and simpler to install. Export is also the step before int8: the int8 ONNX file took 8.4 ms with the same accuracy (0.856 against 0.850, one photo). Its file is 11.3 MB instead of 47 MB.

Check your understanding

MobileNetV3 in ONNX int8 takes 3.4 ms a photo with accuracy 0.869. ResNet-18 in ONNX int8 takes 8.4 ms with 0.856. Hamid's desk sends one photo every few seconds. Which one would you choose for product types, and what would you measure before you switch?

Show the answer

Both are far faster than the desk needs, and 0.869 against 0.856 is about two photos of 160, close to noise. Either works. MobileNetV3 is smaller (4.7 MB), so it is a fair choice. Before switching, measure it on the target computer with one thread, and measure the accuracy on each class and slice, not only overall. Also check its licence and training data, as for ResNet-18.

Recap

  • Measure two things after every change: the time per photo on the target computer and the accuracy on the same test photos.
  • “Small” is not “fast”: MobileNetV3 took 84.9 ms in PyTorch on one thread, and 3.4 ms in ONNX int8.
  • ONNX fp32 was slower than PyTorch on one thread (17.1 against 10.2 ms), but it removes PyTorch from the service. int8 gave 8.4 ms, 11.3 MB and the same accuracy (0.856).
  • A batch of 16 was slower per photo on one thread. Smaller images saved time and cost about 2 photos of 160.
  • The service’s small PatchCore in int8: 2.9 MB, test AUROC 0.836 against 0.847 in 32 bits: about the same quality, without PyTorch.
  • vision inspect judges photos in about 21 ms each after a slower first call.

The inspector is now fast and light. Next, Amira serves it as a small web service. It answers with a confidence and falls back to a person when it cannot judge. Its review folder gives new labels without bias.

Next: The image service and its review queue