Skip to content
Français

Domain shift

  • Lesson 27
  • 40 min
  • Module 8: Run vision in production

Where we are

Almost every method so far was tested on photos from one studio and one camera. Larkfield's world changes: the catalogue moves to a new studio, and the returns desk gets a new camera. In this lesson, we see a good inspector fail on every photo, find the shift in three places, and try the cheapest fixes.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Notice when the photos change under a model, before the cost does. A model learned from one studio and one camera. When the light or the camera changes, the model can fail on every photo, even though nothing in the model changed. You will find the shift in the photos, in the scores and in the decisions, and choose a fix.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook compares the old and the new studio and plots the recorded inspector scores of a month of returns. Then it measures the move with the population stability index. It downloads no model and runs in seconds. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer (for the project steps; the notebook needs none of it)

  • The labs repository with the course’s data and models, and your larkvision folder beside it, from Meet the Larkfield images.
  • Your larkvision project at the end of Module 7 (snapshot C42/snapshots/end-of-m07), with its .venv environment.

Tested on macOS with Python 3.12.13, ONNX Runtime 1.31.1 and OpenCV 5.0.0. The Windows commands are written but were not run.

A worker at the returns desk checks candles by eye every day. One morning she puts on new glasses with a yellow tint. Every white candle now looks yellow to her. If she judges by colour, she sends every candle to the outlet. The candles did not change; her eyes did.

A vision model has the same problem. It learned what “normal” looks like from one set of photos. When the camera, the light or the products change, the photos change. This is a domain shift: the new photos come from a different “domain” than the photos the model learned from.

The problem: a good inspector fails every photo

Section titled “The problem: a good inspector fails every photo”

In March, Larkfield’s returns desk replaced its camera on day 16. Nobody told Priya. The new camera gives cooler, darker, softer and noisier photos. In this course, the change was made by our code, so we know exactly when it happened.

Four photos. First, a purple and red bowl on a white table in cool light. Second, the same bowl in warmer, brighter, yellow light. Third, four white tea lights on a dark cloth, sharp. Fourth, the same kind of tea lights, greyer, softer and with visible noise.
Left: catalogue test photo P-a1f5a57a in the old studio and in the new studio (photo: Ryan Baumann, CC BY 2.0, via Open Images; the warmer light is synthetic, made by our code). Right: returns photos of day 14 (R-228fd50e) and day 17 (R-3e8b375c) (VisA dataset, Amazon, CC BY 4.0; the new camera on day 17 is synthetic, made by our code).
Text description of the diagram

Four photos in a row (two by two on a phone). The first two show the same bowl: in the old studio the wall is light grey; in the new studio the whole photo is warmer and more yellow. The last two show four tea lights from above on a dark cloth. On day 14 the candles are sharp and white. On day 17, after the camera change, they are grey, a little blurred, and the photo has grain.

To a person, the day 17 candles are still clearly good candles. To the inspector, they are not. The small PatchCore from Module 6 compares every patch of a photo with a memory of good patches. Every patch of a new-camera photo is far from that memory. So every photo gets a high score.

  • Days 1 to 15 (old camera)
  • Days 16 to 30 (new camera)
The recorded scores of the small PatchCore (the 32-bit version of Module 6) for every photo of the month, by day. The line is its fail line, 10.61, chosen on the validation photos. The camera changed on day 16.
Text description of the chart

A scatter plot with the day of the month (1 to 30) across and the anomaly score up, and a horizontal line for the fail line at 10.61. From day 1 to day 15, most dots lie between 5 and 10, below the fail line, with a few dots above it. From day 16, every dot jumps up to about 13 to 15, above the fail line. Three dots stand far above the rest, at about 29 to 31 on days 5, 19 and 29. These are the photos with a face, which the last lesson of this module explains. The mean score is 7.65 on days 1 to 15 and 14.21 on days 16 to 30.

The mean score goes from 7.65 to 14.21. On days 16 to 30, all 60 photos fail. With the course’s costs, the inspector now costs 14,560 dollars a day: almost every good candle goes to the outlet. In the first week, the same inspector cost 994.3 dollars a day.

The model did not get worse at seeing damage. Its world changed.

A shift shows in three places, from the earliest to the closest to money.

Three boxes side by side. 1. The photos: can a classifier tell old photos from new (a domain classifier)? 0.5 means no shift, 1.0 a clear shift. 2. The scores: did the scores move (PSI against the validation week)? Above 0.25 is a large move. 3. The decisions: did the share of fail or check move (the cost alarm)? An alarm at 10 points above its validation level. All three lead to one orange box: an alarm means stop and look at the photos in review before you change anything.
Three places to see a domain shift. The photos show it first, without any model output. The decisions are closest to money.
Text description of the diagram

Three boxes in a row (stacked on a phone). The first, “The photos”, asks whether a classifier can tell old photos from new ones: 0.5 means no shift, 1.0 a clear shift. The second, “The scores”, asks whether the scores moved, measured with PSI against the validation week: above 0.25 is a large move. The third, “The decisions”, asks whether the share of fail or check moved: an alarm at 10 points above its validation level. Arrows from all three go to an orange box: “An alarm: stop and look at the photos in review/ before you change anything”. A note below: the photos show a shift earliest; the decisions are closest to money.

Put the old photos in one pile and the new photos in another. Train a classifier to guess the pile of each photo from its embedding (the numbers a network makes from a photo, from Image embeddings). This is a domain classifier. If it cannot do better than a coin, the AUROC is about 0.5: no shift it can see. If it always guesses right, the AUROC is 1.0.

This works on photos alone, before any label or score. Priya measured it with cross-validation:

Two piles of photos Domain classifier AUROC Reading
Two halves of the catalogue test photos 0.472 no shift (a coin)
Returns: validation photos against days 1–15 0.529 no shift
Catalogue test photos against the new studio 0.838 a clear shift
Returns: days 1–15 against days 16–30 1.000 a complete shift

The first two rows are the control: two piles from the same place give about 0.5. Always measure a control. It shows what “no shift” looks like on your photos.

2. The scores: the population stability index

Section titled “2. The scores: the population stability index”

The population stability index (PSI) compares two distributions of one number. Here, the number is the anomaly score. We cut the scores of the good validation photos into bins with equal shares. Then we count which share of this week’s photos falls into each bin. If the shares stay the same, the PSI is near 0.

For each bin, PSI adds: (new share − old share) × ln(new share ÷ old share). The usual reading: under 0.1 is small, 0.1 to 0.25 is moderate, above 0.25 is large.

A tiny example with four bins, each with 25% of the validation scores:

Bin Old share New share (new − old) × ln(new ÷ old)
A (lowest scores) 0.25 0.10 −0.15 × ln 0.4 = 0.137
B 0.25 0.20 −0.05 × ln 0.8 = 0.011
C 0.25 0.30 0.05 × ln 1.2 = 0.009
D (highest scores) 0.25 0.40 0.15 × ln 1.6 = 0.071
PSI 0.23

The scores moved up a little: a moderate move, under 0.25. Now the camera month. With ten bins, all photos of days 16–30 land in the top bin. Nine bins go from 10% to almost 0, and one bin from 10% to 100%. That gives 8.28, the largest value this PSI can reach. The notebook computes it: about 0.20 for days 1–15 and 8.28 for days 16–30.

Hamid does not read PSI. He reads money. In Module 6, the lines sent about 4% of the validation photos to fail and 16% to check. If this week’s share of fail or check is more than 10 points above its validation level, something changed. This is the cost alarm. Each week has about 35 photos in the replay, so a share of one week is less noisy than a share of one day. A day has about 5 photos, and one photo moves a daily share by 20 points.

The project gets the image service in this module. Its command vision replay sends the month’s 135 photos through the service’s inspector, week by week, and prints the three kinds of alarm. The service uses the small PatchCore in 8-bit numbers (int8, the next lesson). So its scores and lines differ a little from the 32-bit version in the chart above.

week photos pass check fail faces review mean score PSI
days 01-07 35 27 5 3 1 6 8.177 0.19
days 08-15 40 34 4 2 0 4 7.394 0.45
days 16-23 32 0 0 32 1 1 14.455 8.28
days 24-30 28 0 0 28 1 1 14.458 8.28
DRIFT ALARM days 08-15: scores moved (PSI 0.45 > 0.25). Look at the photos in review/ before you change anything.
DRIFT ALARM days 16-23: scores moved (PSI 8.28 > 0.25). Look at the photos in review/ before you change anything.
COST ALARM days 16-23: 100% of photos fail, validation level 4%. Stop the automatic decisions and ask Hamid's team to check by hand.
DRIFT ALARM days 24-30: scores moved (PSI 8.28 > 0.25). Look at the photos in review/ before you change anything.
COST ALARM days 24-30: 100% of photos fail, validation level 4%. Stop the automatic decisions and ask Hamid's team to check by hand.

Read it in three steps:

  • Days 16–30: PSI 8.28 and 100% fail. Both alarms fire. The fix is not a new line: the photos are different.
  • Days 8–15: a drift alarm (PSI 0.45), but no cost alarm. The camera was the same. This week had only 40 photos, and 7 of them were damaged, more than usual. A small sample moves PSI. This is a false alarm, and the lesson is to look before you act. The alert budget in the anomaly detection course explains why small samples alarm more often.
  • No check after the change: the review folder gets almost nothing on days 16–30, because the inspector is sure: everything fails. A shift can make a model confident and wrong. That is why the cost alarm watches the fail share too, not only the checks.

The catalogue’s new studio: accuracy by slice

Section titled “The catalogue’s new studio: accuracy by slice”

The catalogue team also moved to a new studio with warmer light. Here we have labels, so Priya can measure the accuracy on each slice (a part of the photos with one shared property, here the studio). Slice analysis in the machine learning course explains slicing in detail.

Method (catalogue test, 160 photos) Old studio New studio
Colour histogram + 5 nearest neighbours 0.263 0.225
HOG + logistic regression (no colour) 0.356 0.356
ResNet-18, last layer only 0.812 0.794
CLIP linear probe 0.894 0.900

The colour method loses about 4 points, and ResNet-18 about 2. HOG loses nothing, because it has no colour. CLIP’s probe does not move: a difference of one photo is noise (one photo is 0.6 points). So the same shift breaks one method and not another. Measure each method on each slice; do not assume.

There are three kinds of fix:

  1. A camera rule. Fix the camera’s white balance and exposure, or correct each photo by code. Grey world is a simple rule: scale each colour channel so that the photo’s average colour is grey. It removes a colour cast.
  2. New “normal” photos. For an anomaly model, add good photos from the new camera to its memory. For a classifier, add labelled photos from the new domain. Or augment the training photos: change their colour, light and blur by code, so the model learns to ignore them.
  3. New labels and new lines. Collect and label photos from the new domain, then choose the lines again on a new validation set.

Priya measured the first two on days 16–30 with the small PatchCore. For the second fix, a person checked 30 good photos of the new camera, and they went into the memory. So the cost is measured on the other 30 photos of days 16–30, 4 of them damaged.

Cost per day of the small PatchCore (32-bit) on days 16 to 30 after each fix, with lines chosen on the validation photos and the course's cost assumptions. None: no fix. Grey: the grey-world rule. 30 new: 30 new-camera photos in the memory. Week 1: days 1 to 7 (old camera, no fix), for comparison.
Text description of the chart

A column chart with four columns. No fix: 14,560 dollars a day. Grey-world rule: 13,460. Thirty new-camera photos in the memory: 3,830. For comparison, days 1 to 7 with the old camera: 994.3.

Fix on days 16–30 AUROC Good items lost (of 26) Damaged sold as new (of 4) Cost a day
No fix 0.49 26: all 0 14,560
Grey-world rule 0.538 24 0 13,460
30 new-camera photos in the memory 0.683 2 3 3,830

On the catalogue’s new studio, the same rule helps. It lifts the colour histogram from 0.225 to 0.281, because warmer light is only a colour cast. On the returns, grey world does almost nothing: the new camera is not only colder, it is also darker, softer and noisier. Thirty new photos help a lot, but now 3 of the 4 damaged candles pass, and the AUROC of 0.683 is weak. The lines were chosen for the old camera, so they do not fit the new scores.

Neither fix is enough. Priya’s advice to Hamid for day 16 is:

  1. Stop the automatic decisions. His team checks by hand for a few days.
  2. Collect good photos with the new camera: a few hundred, checked by a person, for the memory.
  3. Label damaged photos from the new camera and choose the lines again on them, as in Module 6.
  4. Next time, test the camera first. A camera change is a planned event. Take 50 photos with the new camera before the switch, and run the replay on them.

The snapshot end-of-m08 adds the whole image service to end-of-m07: the inspector, the drift checks, the review folder, the face blur, and three commands (vision inspect, vision serve, vision replay). Eight tests come with them: 32 in all. The other lessons of this module use the same snapshot.

The heart of the drift check, in larkvision/drift.py:

def psi(expected, actual, bins=10):
"""Population stability index of `actual` against `expected` (bins from the quantiles of `expected`)."""
e = np.asarray(expected, float)
a = np.asarray(actual, float)
edges = np.unique(np.quantile(e, np.linspace(0, 1, bins + 1)))
edges[0], edges[-1] = -np.inf, np.inf
pe = np.histogram(e, edges)[0] / len(e)
pa = np.histogram(a, edges)[0] / len(a)
pe, pa = np.clip(pe, 1e-4, None), np.clip(pa, 1e-4, None)
return float(((pa - pe) * np.log(pa / pe)).sum())
def business_alarms(shares, reference, margin=0.10):
out = []
for zone in ("fail", "check"):
if shares[zone] > reference[zone] + margin:
out.append(f"{shares[zone]:.0%} of photos {zone}, validation level {reference[zone]:.0%}")
return out

The bins come from the validation scores, so each bin holds 10% of them. np.clip keeps an empty bin from dividing by zero. That small floor is why the largest PSI here is 8.28, not infinity.

This step replaces the files of your project with the snapshot’s files. If you changed a file yourself, keep a copy first.

  1. Open a terminal in your larkvision folder, next to pyproject.toml. The folder nextia-learning-labs is beside it.

  2. Activate the environment. On macOS and Linux, run source .venv/bin/activate. On Windows (PowerShell), run .venv\Scripts\Activate.ps1.

  3. Copy the snapshot’s code, tests, package list and README into your project. On macOS and Linux:

    Terminal window
    S=../nextia-learning-labs/C42/snapshots/end-of-m08
    cp -R "$S/larkvision" "$S/tests" "$S/requirements.txt" "$S/README.md" .

    On Windows (PowerShell):

    Terminal window
    $S = "..\nextia-learning-labs\C42\snapshots\end-of-m08"
    Copy-Item -Recurse -Force "$S\larkvision", "$S\tests", "$S\requirements.txt", "$S\README.md" -Destination .
  4. Install the new packages (FastAPI, ONNX Runtime, OpenCV and three more):

    Terminal window
    pip install -r requirements.txt
  5. Run the tests:

    Terminal window
    python -m pytest -q

    The last line is 32 passed. If it says 24 passed, the new test files are missing.

  6. Tell the project where the data and the models are. The service needs the models folder too. On macOS and Linux:

    Terminal window
    export LARKVISION_DATA="$(cd ../nextia-learning-labs/C42/data && pwd)"
    export LARKVISION_MODELS="$(cd ../nextia-learning-labs/C42/models && pwd)"

    On Windows (PowerShell):

    Terminal window
    $env:LARKVISION_DATA = (Resolve-Path ..\nextia-learning-labs\C42\data).Path
    $env:LARKVISION_MODELS = (Resolve-Path ..\nextia-learning-labs\C42\models).Path
  7. Replay the month:

    Terminal window
    vision replay

    The first line gives the model, its lines, and the time per photo on your computer. The table and the alarms are the ones above. The last lines are:

    Daily share of fail (validation level 4%) and check (16%), about 5 photos a day:
    01:0% 02:0% 03:20% 04:0% 05:20% 06:0% 07:20% 08:20% 09:0% 10:0% 11:0% 12:0% 13:20% 14:0% 15:0% 16:100% 17:100% 18:100% 19:100% 20:100% 21:100% 22:100% 23:100% 24:100% 25:100% 26:100% 27:100% 28:100% 29:100% 30:100%
    Wrote outputs/replay.csv and 12 photos to review/ (faces blurred).

The two variables last only as long as the terminal. In each new terminal, activate the environment and set both variables again. If a command says no data folder found or no models folder found, a variable is missing. On the course’s laptop, the replay took about 3 seconds.

Read the daily line: before day 16, a day’s fail share jumps between 0% and 20%, because one photo of five is 20%. A daily alarm at 10 points would fire on noise. The weekly alarm does not.

When to watch for a shift, and what it costs

Section titled “When to watch for a shift, and what it costs”

Watch every model that sees photos from the world: cameras, light, seasons, packaging and products all change. Check the photos with a domain classifier whenever something is planned to change (a new camera, a new studio, a new supplier). Watch the scores and the zone shares every week.

Do not retrain on every alarm. Look at the photos first. Week 2’s alarm was noise.

What it costs: a small sample of new photos for each check, and a person’s time to look at the review folder after an alarm. When the shift is real, it also costs new labels. Data and concept drift in MLOps and LLMOps: Operating AI in Production covers drift monitoring for a whole service in depth.

Guided practice: the shift in the notebook

Section titled “Guided practice: the shift in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The data cell ends with a Ready: line.
  3. In section 1, compare the two bowls and the mean colours of the two studios. The new studio is warmer: more red, less blue.
  4. In section 2, predict, then plot the month’s scores. You see the jump on day 16.
  5. In section 3, run the PSI cell. You see about 0.2 for days 1–15.

Before you run it, predict: is the PSI of days 16–30 just above 0.25, or far above? In the notebook, section 4, put it in psi_late, then run the check cell.

Show the answer
late = month[month["day_number"] >= 16]["score"]
psi_late = psi(reference, late)
print(round(psi_late, 2))
8.28

Far above. Every photo of days 16–30 scores above every good validation photo, so all of them land in the top bin. That is the largest PSI with ten bins: no larger number could say “completely different” more clearly.

Common mistake

Moving the line instead of fixing the photos

Symptom: on day 17, Tomás sees 100% fail and raises the fail line from 10.6 to 16 so that “the numbers look normal again”. The fail share drops.

Cause: the scores went up because every patch looks unusual, not because the candles are damaged. With the new camera, good and damaged candles get mixed scores: the AUROC on days 16–30 is 0.49, a coin. A new line on a coin still sorts at random.

Fix: stop the automatic decisions, collect good photos with the new camera, then choose the lines again on photos from that camera.

Consequence in AI work: a line moved to silence an alarm hides the shift. Damaged candles go back on sale as new, and the next alarm never fires.

Check your understanding

A domain classifier separates last month's returns photos from this month's with an AUROC of 0.51. Two halves of last month give 0.49. What do you conclude, and what do you still need to check?

Show the answer

The classifier tells the two months apart no better than two halves of the same month. So there is no shift that the embeddings can see. The control (0.49) shows what “no shift” looks like on these photos. You still need to watch the scores and the decisions. A change that the embedding ignores, such as a small new kind of damage or a new product, can still move the zone shares.

Check your understanding

With four equal bins, the old shares are 0.25 each and the new shares are 0.25, 0.25, 0.10 and 0.40. Compute the PSI and say whether it is small, moderate or large.

Show the answer

Only bins C and D change. C: (0.10 − 0.25) × ln(0.10 ÷ 0.25) = −0.15 × −0.916 = 0.137. D: (0.40 − 0.25) × ln(0.40 ÷ 0.25) = 0.15 × 0.470 = 0.071. A and B add 0. PSI = 0.21: a moderate move, under 0.25. With a small sample, look at the photos before you act.

Check your understanding

The replay shows a drift alarm in days 8–15 (PSI 0.45) and both a drift alarm and a cost alarm in days 16–30. Why is the first one probably a false alarm, and why does the second one need action?

Show the answer

Days 8–15 used the same camera. The week had only 40 photos and 7 damaged ones, more than usual, so the score distribution moved by chance. The zone shares stayed near their validation levels, so no cost alarm fired. In days 16–30, every photo fails (100% against a level of 4%) and PSI is at its largest. The camera changed, the inspector sorts at random (AUROC 0.49), and it costs about 14,560 dollars a day. Stop the automatic decisions, collect good photos with the new camera, and set the lines again.

Recap

  • A domain shift is a change in the photos (camera, light, products), not in the model. The new returns camera made the inspector fail all 60 photos of days 16–30: 14,560 dollars a day.
  • Watch three places: the photos (a domain classifier: 0.5 no shift, 1.000 for the camera change), the scores (PSI: 8.28 for the camera change) and the decisions (the cost alarm: 100% fail against a level of 4%).
  • Measure a control and expect false alarms from small samples: week 2’s PSI of 0.45 was noise.
  • Measure each method on each slice. The new studio cost the colour histogram 4 points, and HOG and CLIP nothing.
  • The cheap fixes were not enough: grey world 13,460 dollars a day, 30 new-camera photos 3,830. Stop, collect good photos with the new camera, and set the lines again.
  • vision replay runs the month through the service and prints the alarms.

You can now see when the photos change under a model. Next, the model must also be fast enough on the warehouse’s ordinary computer. You change the image size, export to ONNX and use 8-bit numbers, and measure the time and the accuracy after each step.

Next: Fast enough on a CPU