Skip to content
Français

People in images

  • Lesson 30
  • 35 min
  • Module 8: Run vision in production

Where we are

Our service sends unsure photos to a person and keeps a copy for them. Some photos need care for another reason: a person leans over the table, and their face is in the photo. In this lesson, the service blurs faces by default and keeps as little as it can, and we see why this course does not build vision about people.

What you will be able to do

Keep people out of a system that was built for products. A returns photo can catch a worker’s face by accident. The service must not store, copy or share that face. You will blur faces before any copy is kept and choose the face finder’s threshold from the cost of each mistake. You will keep as little as possible, and learn which tasks about people this course does not build, and why.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook finds the three painted faces in the month’s photos with the service’s face finder and blurs them. It checks the blur with a test, and measures what the threshold does on the catalogue. Its setup cells install OpenCV and download the returns and catalogue photos and the face finder (31 MB). About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer (for the project steps; the notebook needs none of it)

  • Your larkvision project at the end of Module 8 (snapshot C42/snapshots/end-of-m08), set up in Domain shift, with its .venv environment.

Tested on macOS with Python 3.12.13 and OpenCV 5.0.0 (opencv-python-headless). The Windows commands are written but were not run.

You take a photo of a parcel at the post office for a claim. A stranger stands behind you, and her face is in the corner of the photo. You send the photo to the shop. Now her face sits in the shop’s claims system. She never agreed to that, and she does not know it.

The same happens at Larkfield’s returns desk. The camera points at the table. A worker leans over the table to move a candle, and the camera takes the photo at that moment.

In the month that vision replay runs, three photos have a face in the background. In this course, the faces come from three public-domain paintings by Vermeer, van Gogh and Leonardo. Our code pasted them over a corner of the table. No photo in this course shows a real person.

Top row: three photos of four tea lights with a painted face in a corner, partly over a candle: Vermeer's Girl with a Pearl Earring, van Gogh's self-portrait, and Leonardo's Mona Lisa. Bottom row: the same three photos as the service keeps them, with each face blurred beyond recognition into soft coloured blocks; the candles are unchanged.
Top: the month's photos R-fcd0915d, R-fd29f7cf and R-e382ce16, each with a face from a public-domain painting added by our code (Johannes Vermeer, Girl with a Pearl Earring; Vincent van Gogh, Self-Portrait, 1887; Leonardo da Vinci, Mona Lisa; Wikimedia Commons). Bottom: the copies that the service keeps in its review folder. Candle photos: VisA dataset, Amazon, CC BY 4.0; the last two also have the new camera, made by our code.
Text description of the diagram

Two rows of three photos (pairs on a phone). In the top row, each photo shows four tea lights from above. In the corner of each photo, over the edge of one candle, there is a painted face: the girl with a pearl earring, van Gogh with his red beard, and the Mona Lisa. The bottom row shows the same three photos after the service: each face is now a soft patch of colour blocks, and nothing of the face can be recognised. The rest of the photo is unchanged.

A face in a returns photo causes two problems:

  • The inspector fails the item. A face is far from every patch of a good candle. All three photos score about 29, far above the fail line of 10.826. All three candles were good: the course’s answer key says so (the month’s photos have no labels in your data). Without a person, three good candles go to the outlet.
  • The photo would be copied. A failed or unsure photo goes to the review folder. From there it goes to a labelling tool, a backup and maybe a training set. Each copy keeps the face.

Blur by default means: before the service writes any copy of a photo, it looks for faces and blurs them. Nobody has to remember to do it. Keep less means: the service keeps only what a person needs for the decision, and only for as long as they need it.

To blur a face, the service must first find it. A face detector (here called a face finder) draws a box around each face and gives it a score from 0 to 1. It does not know who the person is: that would be face recognition, which this course does not build. The service uses YuNet, a small face detector from OpenCV Zoo (MIT licence, 230 KB).

The service blurs each box with a margin of 30% around it, which also covers hair and ears. It makes the region 12 times smaller, enlarges it again, and smooths it. Then it checks the result with a test, not by eye: the face finder must find nothing in the blurred copy.

The face finder keeps a face when its score is at or above a threshold. The two mistakes have very different prices:

  • A missed face stays in the review folder and every copy of it. That harms a person, and no refund repairs it.
  • A false face (a bowl or a reflection called a face) is blurred for nothing. The photo goes to a person: about one minute, 0.50 dollars.

So the threshold should be low enough to find every face, even if it blurs some bowls. Priya measured the face finder on three sets of photos at four thresholds. The catalogue has no person at all: four filters and a look at every photo removed them. So every “face” found in the catalogue is a false face.

Threshold Painted faces found (of 3) Returns photos flagged (of 735, no person) Catalogue photos flagged (of 733, no person)
0.9 2 (van Gogh missed: 0.796) 0 0
0.6 (the service) 3 0 14
0.3 3 0 83
0.1 3 (+ 4 false faces) 11 314
Catalogue photos (733, with no person in them) where the face finder found a face, at four score thresholds. The service uses 0.6.
Text description of the chart

A column chart with four columns. Threshold 0.9: 0 photos. Threshold 0.6, the service’s: 14. Threshold 0.3: 83. Threshold 0.1: 314.

At 0.9, the face finder misses a face: too strict. At 0.6, it finds all three faces and flags no returns photo without a person. On the varied catalogue photos, it calls 14 bowls, mugs and candles a face. The threshold that works depends on the photos, as every line in this course does. On the desk’s photos, 0.6 costs almost nothing, and the service uses it.

Three catalogue photos, each with a red box drawn by the face finder: an amber glass bowl on a stand with a large box around its top, a burning candle with a tall thin box at the right edge, and a blue and white ceramic bowl with a box around the whole bowl. None of them shows a person.
False faces at threshold 0.6: catalogue photos P-30f3b3b0 (photo: Pattie Hite), P-1d9c3686 (Riza Nugraha) and P-5c6384bd (Robert Couse-Baker), all CC BY 2.0 via Open Images. The red boxes were drawn by our code from the face finder's output.
Text description of the diagram

Three photos in a row (two on a phone). The first is an amber glass bowl on a stand, in front of a patterned cloth; a red box covers the top of the bowl. The second is a lit candle in the dark; a tall thin red box sits at its right edge. The third is a blue and white glazed bowl on white; a red box covers most of the bowl. No photo shows a person.

Amira wrote down what the service keeps, and for how long.

Four boxes in a row. The photo that arrives: in memory only, for the length of the request, never written to disk. A copy in review: only when a person must look, every face blurred first. The answer and the log: item, zone, score, reasons, number of faces; no face, no image. Delete: review copies after the check, outputs after the report.
What Amira's service keeps, from the photo that arrives to the copies it deletes.
Text description of the diagram

Four boxes in a row (stacked on a phone), joined by arrows. “The photo that arrives” stays in memory only for the length of the request and is never written to disk. “A copy in review/” is written only when a person must look, with every face blurred first. “The answer and the log” hold the item, the zone, the score, the reasons and the number of faces, but no face and no image. “Delete”: the review copies after the check, and the outputs after the report.

The code that blurs, in larkvision/privacy.py:

def blur(img, boxes, margin=0.3):
"""A copy of the photo with each box (x, y, w, h) and a margin around it blurred beyond recognition."""
out = img.convert("RGB").copy()
for x, y, w, h in boxes:
m = int(margin * max(w, h))
x0, y0 = max(0, x - m), max(0, y - m)
x1, y1 = min(out.width, x + w + m), min(out.height, y + h + m)
if x1 <= x0 or y1 <= y0:
continue
region = out.crop((x0, y0, x1, y1))
small = region.resize((max(1, region.width // 12), max(1, region.height // 12)), Image.BILINEAR)
out.paste(small.resize(region.size, Image.NEAREST).filter(ImageFilter.GaussianBlur(4)), (x0, y0))
return out

And in larkvision/review.py, the one line that writes a copy always blurs first:

blur(img, faces).save(folder / f"{item_id}.jpg", quality=90)

There is no other line in the service that writes a photo. The original photo is never saved. A test, test_a_face_sends_the_photo_to_review_blurred, blurs a face box on a photo of random noise. It checks that the detail in that region falls to less than a third. A second test checks that the rest of the photo does not change.

  1. Open a terminal in your larkvision folder and activate the environment (source .venv/bin/activate, or .venv\Scripts\Activate.ps1 on Windows).

  2. In a new terminal, set LARKVISION_DATA and LARKVISION_MODELS again, with the commands of step 6 in Domain shift.

  3. Run the privacy tests:

    Terminal window
    python -m pytest -q tests/test_service.py

    The last line is 4 passed.

  4. Judge two month photos with a painted face. On macOS and Linux:

    Terminal window
    vision inspect ../nextia-learning-labs/C42/data/returns/month/R-fcd0915d.jpg ../nextia-learning-labs/C42/data/returns/month/R-fd29f7cf.jpg

    On Windows, write the paths with ..\nextia-learning-labs\C42\data\returns\month\.

    R-fcd0915d: fail score 29.463 confidence 1.00 34 ms -> review: a person in the photo (1 face(s), blurred)
    R-fd29f7cf: fail score 29.929 confidence 1.00 21 ms -> review: a person in the photo (1 face(s), blurred)
  5. Open review/today/R-fcd0915d.jpg in an image viewer. The face is blurred. The original in the labs folder is unchanged.

  6. Delete the review folder when you are done: vision reset removes outputs/ and review/.

The zone is “fail” with confidence 1.00, but the photo still goes to a person, because of the face. That is on purpose: the inspector’s answer on a photo with a person in it is not to be trusted, so a person decides. In vision replay, the three faces are found and blurred, one in each of weeks 1, 3 and 4.

A face finder misses faces that are turned away, small, blurred by motion, or partly hidden. It finds no bodies, no hands, no name badges and no licence plates. So blur by default is the last guard, not the only one:

  • Point the camera at the table. Hamid’s team points the desk camera at the tray only. Then fewer people enter the photo at all.
  • Crop to the product. The inspector needs only the tray area. Cut the photo to it before anything else.
  • Tell people. A sign at the desk says that a camera photographs returned items, and why.
  • Limit who opens the review folder, and delete its copies after the check.

Some vision tasks are about people themselves. They guess who a person is (face recognition) or what they feel (emotion recognition). Or they guess their age, gender or ethnic group (attribute inference). This course does not build any of them, for three reasons.

They fail unevenly. A model that guesses something about a person is often much less accurate for some groups than for others. A well-known 2018 study, Gender Shades (Buolamwini and Gebru), reported that commercial gender classifiers made far more errors on darker-skinned women than on lighter-skinned men. A returns inspector that errs costs a candle. A system that errs about a person can cost them a job or their freedom.

Some of them claim more than a photo can show. A face shows an expression, not a feeling. Emotion and character cannot be read reliably from a face. A model trained to do it learns the labellers’ guesses.

The law treats them differently. In many places, data that identifies a person from their face is a special kind of personal data. Strict rules apply to it. The EU AI Act, Article 5 (1) (f), forbids AI systems “to infer emotions of a natural person in the areas of workplace and education institutions”, except for medical or safety reasons. Rules differ between countries and change. This is not legal advice: ask your legal team before any project that looks at people.

So Larkfield’s rule is simple. The returns service looks at products. When a person appears, the service blurs them, sends the photo to a person, and does not learn from the face. Minimize and retain deliberately and Bias and accessibility in AI Security, Privacy and Responsible Design go deeper. They cover keeping less data and harm to groups.

Blur by default in every system whose photos can catch people by accident. Examples are returns desks, warehouses, shelves, delivery photos and street photos (faces and licence plates).

Do not rely on the blur alone. Point the camera away from people, crop, delete early, and limit access.

What it costs: a face finder on every photo, a 230 KB model that took about 2 ms a photo here. A few photos go to a person for nothing (false faces). And a test must check the blur. All this is small next to the cost of a face in a training set that is copied for years.

Guided practice: the faces in the notebook

Section titled “Guided practice: the faces in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The data cell ends with a Ready: line.
  3. In section 1, find the faces. You see 3 photos with a face and the three paintings.
  4. In section 2, blur them. You see the three blurred copies.
  5. In section 3, check the blur. For each photo you see faces found after the blur: 0, and the detail in the face box falls by about half: for example 54.7 -> 29.7.
  6. In section 4, predict, then run the cell. You see 14 catalogue photos with a false face and 2 of the 3 painted faces found at 0.9.

Change one thing: the threshold, to 0.3. Before you run it, predict: about how many of the 733 catalogue photos get a false face? In the notebook, section 5, put the number in false_03, then run the check cell.

Show the answer
false_03 = len(flagged(face_finder(0.3), catalogue["file"]))
print(false_03)
83

About six times more than at 0.6, and no more faces found: 0.6 already found all three. On the desk’s photos, 0.3 would flag no extra photo (0 of 735 returns photos). On varied photos, each false face costs a person a minute. Keep 0.6 for the desk, and measure again when the photos change.

Common mistake

Blurring after the copy

Symptom: Tomás adds a feature: “save every photo to raw/ for the next model, then blur the copy in review/”. A month later, the face of a worker is found in a training set that was shared with a supplier.

Cause: the original was written to disk before the blur. Every backup and every copy of raw/ kept the face. Blurring the review copy did not protect the first copy.

Fix: blur before the first write, in the one function that writes photos, and never save the original. If a new model needs more photos, collect them through the same function, so they are blurred too. A test checks it.

Consequence in AI work: personal data spreads with every copy. It is easy to stop it before the first copy, and very hard to stop it after.

Check your understanding

At threshold 0.9 the face finder misses one of the three painted faces and finds no false face. At 0.6 it finds all three and flags 14 catalogue photos. Which threshold should the returns service use, and why?

Show the answer

0.6. A missed face is the expensive mistake: it stays in every copy of the photo and harms a person. A false face costs one blur and about one minute of a person’s time. On the returns photos, 0.6 flags no photo without a person, so it costs almost nothing there. Measure again if the camera or the photos change.

Check your understanding

A photo of a good candle with a face in the corner scores 29.5: fail, confidence 1.00. Why does the service still send it to the review folder, and what is in the copy?

Show the answer

A face makes the inspector’s answer untrustworthy. The face is far from every good patch, so a good candle scores like a damaged one. So the service adds the reason “a person in the photo” and a person decides. The copy in the review folder has the face blurred. The original photo is never written.

Check your understanding

Omar suggests adding emotion recognition to the desk camera, 'to see if the workers are tired'. Give two reasons why this course does not build it.

Show the answer

First, a face shows an expression, not a feeling, so the model would learn labellers’ guesses, and such models fail unevenly across groups of people. Second, it is legally risky. The EU AI Act forbids emotion recognition at work, with narrow exceptions. And data that identifies people from their faces has strict rules in many places. Also, the camera exists to inspect products; using it to watch workers changes its purpose. Ask the legal team, and point the camera at the table.

Recap

  • A face in a returns photo makes the inspector fail a good candle (score about 29) and would put the face in every copy of the photo.
  • Blur by default: the service finds faces with YuNet and blurs them before any copy is written. The original is never saved. A test checks that the blur works.
  • Choose the threshold from the prices. At 0.9, a face was missed. At 0.6, all three were found; 14 of 733 catalogue photos got a false face, and 0 of 735 returns photos.
  • Keep less: point the camera at the table, crop to the product, keep only blurred review copies, delete them after the check, and limit access.
  • This course does not build face recognition, emotion recognition or guesses about age and gender. They fail unevenly, claim more than a photo shows, and carry legal risk.

You have finished the teaching modules. You can now take a vision method from a notebook to a service: fast on a CPU, watched for shifts, with a person for unclear cases and no faces in its copies. Next, the case studies: first, visual inspection with few defects on other products.

Next: Visual inspection with few defects