Handle visual uncertainty

What you will be able to do

Find out what a picture does not let the pipeline know, with measurements that do not come from the model’s opinion of itself. Then keep every value that is not available explicit: absent, with the reason, never a plausible guess.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The three statuses and the reasons a field may be missing, from Define the extraction contract.
• OCR scores and line boxes, from Text and OCR paths.
• combine.py: rules first, the model when needed, grounding, from Combine methods.

What you need on your computer

• Your project at the end of the previous lesson: python -m pytest -q prints 81 passed.
• The labs repository in ~/projects/nextia-learning-labs, up to date.

Tested with Python 3.12.13, RapidOCR 3.10.0 and Pillow 12.3.0 on macOS (Apple silicon). The Windows and Linux commands are not tested. The damaged photos and the smaller pictures are made by the course’s generator, on purpose.

Get the lesson’s file

1. In the project folder, copy the module and its test from the snapshot:

   macOS / Linux

   S=../nextia-learning-labs/C13/snapshots/end-of-m04
   cp $S/doc_extract/uncertainty.py doc_extract/
   cp $S/tests/test_uncertainty.py tests/

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C13\snapshots\end-of-m04"
   Copy-Item "$S\doc_extract\uncertainty.py" doc_extract\
   Copy-Item "$S\tests\test_uncertainty.py" tests\

2. Run the tests:

   python -m pytest -q

   The last line is 84 passed. This is the test count of the snapshot end-of-m04.

A good witness

Think of a witness to a car accident at night. A good witness says: “The car was blue. I could not read the plate: it was too far away.” A bad witness gives a plate number that sounds right, because the police officer wants one. Both sound sure. Only one is useful.

A careful officer does not only ask the witness how sure they are. The officer also checks the conditions: it was dark, the witness stood 50 metres away. These facts do not depend on the witness’s opinion of themselves. And the officer writes “plate: not seen, too far” in the report. They do not leave the box empty, and they do not fill it with a guess.

A picture of a receipt has conditions too: blur, a cut edge, a small size. This lesson measures them, and writes “not seen, and why” in the record.

The problem: pictures that lie about themselves

Four pictures in the course’s set cannot be fully read. The picture below shows them, with what OCR measured and what the model said:

Four pictures that the pipeline cannot fully read, with the OCR measurements and the model's own answer. Only one answer is wrong about itself: D047's top is cut off, and the model says that the shop and the date are not printed.

Text description of the diagram

The first card is D046, a dark and very blurred photo. OCR found no characters, and the model said that every field is unreadable. The second is D047, a photo whose top edge cuts through the shop name: OCR read 255 characters with a mean score of 0.99. The model said that the vendor and the date are not printed, with a confidence of 0.99. The third is D045, whose bottom ends at the tax line. OCR read 266 characters with a score of 0.99. The model said that the total and the payment are cut off. The fourth is the scan D022, made four times smaller: 212 by 275 pixels. Its receipt part is enlarged, so that you can see its pixels. OCR read 32 characters with a mean score of 0.60, and the model said that the page cannot be read.

Read the model’s own words for the three damaged photos, next to the fields that the generator really cut off. Start python in the project folder and type:

from doc_extract import documents, vision
for doc in ("D045", "D046", "D047"):
    d = documents.get(doc)
    ex = vision.read(d.file)
    gone = [f"{n} {f.reason} ({f.confidence})" for n, f in ex.fields.items() if f.value is None and n != "shipping"]
    print(doc, "| cut off:", d.truth.get("printed_but_cut_off", "-"), "| model absent:", ", ".join(gone))

D045 | cut off: ['total', 'payment_method'] | model absent: tax cut_off (0.96), total cut_off (0.99), payment_method cut_off (0.98)
D046 | cut off: - | model absent: vendor unreadable (1.0), order_number unreadable (1.0), date unreadable (1.0), currency unreadable (1.0), subtotal unreadable (1.0), tax unreadable (1.0), total unreadable (1.0), payment_method unreadable (1.0)
D047 | cut off: ['vendor', 'date'] | model absent: vendor not_printed (0.99), date not_printed (0.99)

Two answers are honest. On D045, the model says cut_off for the total and the payment. It also says cut_off for the tax, which is half visible at the edge; the rules read it. On D046, it says unreadable for everything. But on D047 it says not_printed for the vendor and the date, at a confidence of 0.99. That is wrong: every Larkfield receipt prints its shop and its date. They were on the paper, above the edge of the photo. An agent who reads “not printed” thinks that the receipt is strange, not that the photo is bad. So the agent does not ask for a new photo.

Smaller pictures show a second problem. The course made six receipt pictures two and four times smaller, and read them again with OCR and with the model. List the model’s answers that are not right at a quarter of the size. In python:

from doc_extract import compare, documents, vision
for doc in ("D020", "D022", "D036", "D038", "D040", "D044"):
    d = documents.get(doc)
    ex = vision.read(d.file, variant="quarter")
    for n, true in d.fields.items():
        f = ex.fields[n]
        if true is not None and not compare.correct(n, f.value, true):
            print(f"{doc} {n:<13} model: {f.value} ({f.status} {f.reason}, confidence {f.confidence}) | true: {true}")

D020 date          model: 2026-06-26 (found , confidence 0.96) | true: 2026-06-25
D022 vendor        model: None (absent unreadable, confidence 0) | true: Larkfield
D022 order_number  model: None (absent unreadable, confidence 0) | true: LK-850924
D022 date          model: None (absent unreadable, confidence 0) | true: 2026-09-02
D022 currency      model: None (absent unreadable, confidence 0) | true: CAD
D022 subtotal      model: None (absent unreadable, confidence 0) | true: 162.00
D022 tax           model: None (absent unreadable, confidence 0) | true: 21.06
D022 total         model: None (absent unreadable, confidence 0) | true: 183.06
D022 payment_method model: None (absent unreadable, confidence 0) | true: Cash
D038 tax           model: None (absent unreadable, confidence 0.85) | true: 8.44
D040 order_number  model: LK-787043 (found , confidence 0.96) | true: LK-787943
D044 order_number  model: None (absent unreadable, confidence 0.7) | true: LK-395245
D044 subtotal      model: 352.95 (found , confidence 0.88) | true: 371.95
D044 shipping      model: 19.00 (found , confidence 0.8) | true: 0.00
D044 tax           model: 55.60 (found , confidence 0.8) | true: 55.70
D044 total         model: 427.55 (found , confidence 0.95) | true: 427.65

On D022, the model says unreadable for every field: the honest answer. On the other pictures, it writes six wrong values as found. D020’s date is one day off, and D040’s order number has one wrong digit. On D044, four amounts are wrong, and they still add up: 352.95 + 19.00 + 55.60 = 427.55. A check that the total adds up would pass. The model cannot always tell when a picture is too small for it.

The idea: measure the conditions, keep the gaps explicit

The model’s account of itself is not enough: it said “not printed” for a cut-off top, and it wrote misreads as found. So the pipeline measures the conditions of each page itself. These are signals of visual uncertainty: facts about the picture and the answers that say a value may not be knowable. uncertainty.py computes them from the OCR lines that the pipeline already has:

• unreadable: OCR found fewer than 40 characters on the page, or the mean score of its lines is under 0.75.
• low_resolution: the text is small. The median height of the OCR lines is under 14 pixels.
• conflict: two pages of the same file show two different values for the order number, the date or the total.
• unverified: the model gave a value that cannot be found on the page (from the previous lesson).
• absent, with a reason: a field is absent because it is cut_off or unreadable. The reason comes from the model, and combine.py keeps it.
• a reason the contract does not allow: a field is absent for a reason that the contract forbids for that field. The vendor may be missing only as unreadable; the date only as cut_off or unreadable. A receipt always prints both.

The second half of the idea is the record. A value that is not available stays explicit: value is null, status is absent, and reason says why. An empty box hides the difference between “not on the receipt” and “lost in the photo”. A guess hides that anything is missing.

A tiny example: four lines of a small picture

Give page_signals four OCR lines, as OCR might read the bottom of a small, blurred picture. Every value is visible: the text, the box and the score. In python:

from doc_extract import uncertainty
from doc_extract.pages import Line, Page
lines = [Line("TOTAL", (40, 300, 120, 312), 0.62), Line("4O5.6l", (300, 300, 380, 312), 0.48),
         Line("Cash", (40, 330, 90, 342), 0.71), Line("Thank you", (40, 360, 160, 372), 0.80)]
s = uncertainty.page_signals([Page("small.jpg", 1, 300, 400, "px", "ocr", lines)])
print("characters", s.chars, "| mean score", s.mean_score, "| text height", s.text_px, "px")
print(s.flags)

characters 24 | mean score 0.652 | text height 12.0 px
['unreadable: 24 characters found, mean score 0.65']

Follow it by hand. The characters: 5 + 6 + 4 + 9 = 24, under 40. The mean score: (0.62 + 0.48 + 0.71 + 0.80) / 4 = 2.61 / 4 = 0.6525, under 0.75. Each box is 12 pixels high, so the text is small too. The page is flagged unreadable, the stronger signal. The project reports one page signal, not both. Note the line 4O5.6l: OCR read a letter O and a letter l. The rules cannot read an amount from it, so no total is invented from this line.

The signals on the course’s pictures

1. The blurred photo:

   python -m doc_extract.uncertainty D046

   D046  IMG_4887.jpg: 0 characters, mean score 0.0, text about 0.0 px high
     unreadable: 0 characters found, mean score 0.00
     vendor: absent (unreadable)
     order_number: absent (unreadable)
     date: absent (unreadable)
     currency: absent (unreadable)
     subtotal: absent (unreadable)
     shipping: absent (unreadable)
     tax: absent (unreadable)
     total: absent (unreadable)
     payment_method: absent (unreadable)
     vendor          (unreadable)   absent
     order_number    (unreadable)   absent
     date            (unreadable)   absent
     currency        (unreadable)   absent
     subtotal        (unreadable)   absent
     shipping        (unreadable)   absent
     tax             (unreadable)   absent
     total           (unreadable)   absent
     payment_method  (unreadable)   absent

   The page is unreadable: OCR found nothing. Every field is absent, and each one keeps the model’s reason, unreadable. An agent knows at once: ask the customer for a new photo.

2. The photo cut at the bottom:

   python -m doc_extract.uncertainty D045

   D045  IMG_4880.jpg: 266 characters, mean score 0.99, text about 44.0 px high
     currency: unverified: the model's value cannot be found on the page
     total: absent (cut_off)
     payment_method: absent (cut_off)
     vendor          Larkfield      found
     order_number    LK-754892      found
     date            2026-06-13     found
     currency        CAD            unverified
     subtotal        358.95         found
     shipping        (not_printed)  absent
     tax             46.66          found
     total           (cut_off)      absent
     payment_method  (cut_off)      absent

   The page reads well: 266 characters, score 0.99, text about 44 pixels high. The signals are about fields. The currency is unverified, as in the previous lesson. The total and the payment are absent because they are cut off. The shipping is absent with the reason not_printed, which is normal for a till receipt, so it gives no signal.

3. The photo cut at the top:

   python -m doc_extract.uncertainty D047

   D047  IMG_4894.jpg: 255 characters, mean score 0.991, text about 47.5 px high
     vendor: absent (not_printed), but the contract allows only: unreadable
     date: absent (not_printed), but the contract allows only: cut_off, unreadable
     vendor          (not_printed)  absent
     order_number    LK-192531      found
     date            (not_printed)  absent
     currency        CAD            found
     subtotal        187.00         found
     shipping        (not_printed)  absent
     tax             24.31          found
     total           211.31         found
     payment_method  Visa           found

   The page reads well too, and the model said “not printed”. But the contract allows only unreadable for the vendor, and only cut_off or unreadable for the date. So the project flags both. The contract catches what the model’s confidence hid. The honest reason for both is cut_off. For the vendor, the contract’s rule is too strict for a cut-off top: a better contract would allow cut_off too. That is an improvement you can make; the course keeps the rule.

4. A low-resolution scan:

   python -m doc_extract.uncertainty D024

   D024  scan0006.jpg: 306 characters, mean score 0.961, text about 13.5 px high
     low_resolution: text is about 14 pixels high
     vendor          Larkfield      found
     order_number    LK-103850      found
     date            2026-09-09     found
     currency        CAD            found
     subtotal        89.00          found
     shipping        (not_found)    absent
     tax             11.57          found
     total           100.57         found
     payment_method  Debit          found

   D024 is a 75 dpi scan. Its text is about 13.5 pixels high, just under the limit of 14, so it is flagged. Yet every value is right. (The shipping’s not_found is the rules’ reason when no line matches, as in Provenance representation: the scan prints no shipping line.) A signal says “look here”, not “this is wrong”. The next module decides what goes to a person.

Run python -m doc_extract.uncertainty --all to see every receipt in one table. Only four of the 54 rows have a signal:

D024  scan_image  0.961  low_resolution: text is about 14 pixels high
D045  photo        0.99  currency: unverified: the model's value cannot be found on the page; total: absent (cut_off); payment_method:
D046  photo         0.0  unreadable: 0 characters found, mean score 0.00; vendor: absent (unreadable); order_number: absent (unreadable
D047  photo       0.991  vendor: absent (not_printed), but the contract allows only: unreadable; date: absent (not_printed), but the co

How small is too small?

Back to the six receipts at full, half and quarter size. The script below makes the smaller copies in out/small/, as the course made them. Then it reads every picture with OCR, raw and prepared, and replays the model’s recorded answer for that size. Save it as small_pictures.py in the project folder:

from pathlib import Path
from PIL import Image
from doc_extract import compare, documents, fields, ocr, uncertainty, vision

DOCS = ["D020", "D022", "D036", "D038", "D040", "D044"]
for size, scale in (("full", 1), ("half", 0.5), ("quarter", 0.25)):
    right = {"raw": 0, "prepared": 0, "model": 0}
    printed = 0
    for doc in DOCS:
        d = documents.get(doc)
        path = d.path
        if scale < 1:  # a smaller copy, made as the course made it
            path = Path("out/small") / size / d.file
            path.parent.mkdir(parents=True, exist_ok=True)
            with Image.open(d.path) as img:
                small = img.convert("RGB").resize((round(img.width * scale), round(img.height * scale)), Image.LANCZOS)
            small.save(path, "JPEG", quality=85)
        want = {n: v for n, v in d.fields.items() if v is not None}
        printed += len(want)
        line = f"{size:<8}{doc}"
        for kind in ("raw", "prepared"):
            pages = ocr.read(path, prepare=(kind == "prepared"))
            ex = fields.extract(pages, "ocr")
            k = sum(compare.correct(n, ex.value(n), v) for n, v in want.items())
            right[kind] += k
            line += f" {kind} {k}/{len(want)}"
            if kind == "raw":
                s = uncertainty.page_signals(pages)
        model = vision.read(d.file, 1, variant="" if scale == 1 else size)
        k = sum(compare.correct(n, model.value(n), v) for n, v in want.items())
        right["model"] += k
        print(f"{line} model {k}/{len(want)} | text {s.text_px} px, score {s.mean_score} | {'; '.join(s.flags) or 'no signal'}")
    print(f"{size:<8}all: raw {right['raw']}/{printed}, prepared {right['prepared']}/{printed}, model {right['model']}/{printed}")

Run it. The first run reads 30 pictures with OCR, so it takes about 20 seconds:

python small_pictures.py

full    D020 raw 9/9 prepared 9/9 model 9/9 | text 34.8 px, score 0.997 | no signal
full    D022 raw 8/8 prepared 8/8 model 8/8 | text 21.0 px, score 0.989 | no signal
full    D036 raw 8/8 prepared 8/8 model 8/8 | text 55.6 px, score 0.986 | no signal
full    D038 raw 9/9 prepared 9/9 model 9/9 | text 31.2 px, score 0.995 | no signal
full    D040 raw 8/8 prepared 8/8 model 8/8 | text 82.5 px, score 0.991 | no signal
full    D044 raw 9/9 prepared 9/9 model 9/9 | text 33.7 px, score 0.989 | no signal
full    all: raw 51/51, prepared 51/51, model 51/51
half    D020 raw 9/9 prepared 9/9 model 9/9 | text 19.0 px, score 0.996 | no signal
half    D022 raw 7/8 prepared 7/8 model 8/8 | text 11.0 px, score 0.928 | low_resolution: text is about 11 pixels high
half    D036 raw 8/8 prepared 8/8 model 8/8 | text 29.0 px, score 0.99 | no signal
half    D038 raw 9/9 prepared 9/9 model 9/9 | text 17.0 px, score 0.975 | no signal
half    D040 raw 8/8 prepared 8/8 model 8/8 | text 43.0 px, score 0.992 | no signal
half    D044 raw 9/9 prepared 8/9 model 9/9 | text 18.0 px, score 0.964 | no signal
half    all: raw 50/51, prepared 49/51, model 51/51
quarter D020 raw 7/9 prepared 4/9 model 8/9 | text 9.0 px, score 0.949 | low_resolution: text is about 9 pixels high
quarter D022 raw 0/8 prepared 0/8 model 0/8 | text 6.0 px, score 0.595 | unreadable: 32 characters found, mean score 0.60
quarter D036 raw 8/8 prepared 7/8 model 8/8 | text 15.5 px, score 0.965 | no signal
quarter D038 raw 3/9 prepared 2/9 model 8/9 | text 9.0 px, score 0.716 | unreadable: 121 characters found, mean score 0.72
quarter D040 raw 8/8 prepared 8/8 model 7/8 | text 22.5 px, score 0.98 | no signal
quarter D044 raw 3/9 prepared 2/9 model 4/9 | text 9.0 px, score 0.665 | unreadable: 210 characters found, mean score 0.67
quarter all: raw 29/51, prepared 23/51, model 35/51

• OCR on the raw picture
• OCR on the prepared picture
• The model, prompt v2

Printed values right on six receipts at full, half and quarter size (51 values). At half size, everything still reads; at a quarter, every reader fails, and preparation makes OCR worse.

Text description of the chart

Printed values right, of 51. Full size, for example 1,200 by 1,600 pixels: OCR raw 100%, OCR prepared 100%, the model 100%. Half size, 600 by 800: OCR raw 98%, prepared 96%, the model 100%. Quarter size, 300 by 400: OCR raw 57%, prepared 45%, the model 69%.

At half size, the text is still about 11 to 43 pixels high, and almost every value is right. At a quarter, every reader fails. Preparation makes OCR worse, not better: enlarging a tiny picture cannot add the detail that is not there. The model does best, but it writes six of its errors as found.

Now read the signals in the output. At a quarter, four of the six pictures are flagged unreadable or low_resolution. D044, the picture with four wrong amounts that add up, is flagged unreadable (score 0.67). D040 is not flagged, with its text 22.5 pixels high and a score of 0.98. And the model misread one digit of its order number, while OCR read all eight values right. That is why the pipeline reads with the rules first: here, the model is not called at all.

Pages that disagree

A file with two pages can say two different things. The warranty claim D058 has a form on page 1 and the receipt on page 2, and both show an order number. uncertainty.conflicts compares the order number, the date and the total between pages. Run it on the real claim. Then run it on a copy where page 1 has two digits swapped, as if the customer typed the order number wrong (a constructed case). In python:

import copy
from doc_extract import combine, documents, fields, uncertainty
pages = combine.read_pages(documents.get("D058"))
for p in pages:
    print("page", p.page, p.path, "order", fields.extract([p], "rules").value("order_number"))
print("real:", uncertainty.conflicts(pages))
typo = copy.deepcopy(pages)  # constructed: two digits swapped on the form
for ln in typo[0].lines:
    ln.text = ln.text.replace("LK-921048", "LK-921084")
print("constructed:", uncertainty.conflicts(typo))

page 1 text order LK-921048
page 2 ocr order LK-921048
real: []
constructed: ['conflict: order_number differs between pages: page 1 LK-921084, page 2 LK-921048']

On the real claim, the form and the receipt agree. On the constructed copy, the project names the field and both values with their pages. It does not choose one. The receipt is printed by the till, so it is probably right, but “probably” is a person’s decision.

When to use the signals, and what they cost

• Always compute them. They cost almost nothing: they count characters, scores and heights of OCR lines that the pipeline has already read.
• Read them as reasons to look, not as verdicts. D024 is flagged and right. D040 at a quarter size is not flagged, and the model misread it. The thresholds (40 characters, a score of 0.75, 14 pixels) were chosen on this course’s pictures. Check them on your own documents.
• Keep the reason with every absent value. cut_off asks for a new photo. unreadable asks for a sharper one. not_printed asks for nothing. The contract tells which reasons each field may have.
• Do not trust a reader’s account of itself. The model said “not printed” for a cut-off top at 0.99, and wrote misreads as found at 0.80 to 0.96. Confidence and review routing measures this on all its answers.

Common mistake

A total that looks right

Symptom: A developer sees that D045 has no total and adds one line of code: “if the total is missing, use subtotal + tax”. D045 now shows total 405.61. The record looks complete, and no review is needed.

Cause: The total was computed, not read. 358.95 + 46.66 = 405.61, and nothing on the photo shows it. The course printed the lines of D045’s paper receipt from the generator’s truth. The last six are below the edge of the photo:

LARKFIELD
Home & Garden
88 Queen St W
Toronto ON M5H 2M5
------------------------------------
Date: 2026-06-13 10:31
Order: LK-754892
Transaction: 0522-205
Till 03
------------------------------------
Watering can
1 x 19.95 19.95
Cushion covers (set of 4)
1 x 36.00 36.00
Wooden garden bench
1 x 249.00 249.00
Garden chair
1 x 54.00 54.00
------------------------------------
Subtotal 358.95
HST 13% 46.66
TOTAL CAD 405.61
------------------------------------
Cash 405.61
Thank you! Keep this receipt
for returns and warranty.
Made-up document for a course. Not a real receipt.

This time, the computed value equals the printed one: TOTAL CAD 405.61. But the pipeline could not know that. Below the edge there could be a discount, a gift card, a second tax line or a shipping charge. A computed total that is wrong looks exactly like this one.

Diagnose: Look for values with no evidence. A computed total has no box and no seen text, the text at that place in the evidence record of Provenance representation. In this project, every found value has a line on the page; a total without one is a guess.

Fix: Keep the total absent, with the reason cut_off, as combine.py does. The agent asks the customer for a new photo. Or the agent types the total from the paper receipt as a correction, which Correction workflow records next to the original. Checks such as “the total adds up” belong in validation, where both values are printed: Business validation.

Consequence in AI work: a plausible value is worse than an empty one. An empty value asks a question; a plausible one stops anyone from asking.

Your turn: measure the cut edge

Change one condition: instead of the reasons that the model gives, measure the picture. A cut-off photo has text that reaches the edge of the picture. For every photo and scanned image, compute the empty margin above the highest OCR line and below the lowest one. Give each margin as a share of the picture’s height. Predict which receipts have a margin under 2%. Then check it in python:

from doc_extract import combine, documents
for d in documents.receipts():
    if d.source not in ("photo", "scan_image"):
        continue
    page = combine.read_pages(d)[0]
    if not page.lines:
        print(d.doc_id, d.source, "no text")
        continue
    top = min(ln.box[1] for ln in page.lines) / page.height
    bottom = 1 - max(ln.box[3] for ln in page.lines) / page.height
    mark = " <- text at the edge" if min(top, bottom) < 0.02 else ""
    print(f"{d.doc_id} {d.source:<10} top {top:5.1%}  bottom {bottom:5.1%}{mark}")

Show the answer

D020 scan_image top  2.4%  bottom  6.0%
D022 scan_image top 23.0%  bottom 43.5%
D024 scan_image top 39.2%  bottom 25.7%
D026 scan_image top  5.5%  bottom  3.3%
D028 scan_image top 35.2%  bottom 32.7%
D030 scan_image top 15.1%  bottom 49.9%
D032 scan_image top  3.3%  bottom  4.7%
D033 scan_image top 45.6%  bottom 32.3%
D034 scan_image top 30.0%  bottom 48.0%
D035 photo      top 14.1%  bottom 14.5%
D036 photo      top 10.3%  bottom  9.7%
D037 photo      top 11.5%  bottom 10.4%
D038 photo      top 12.5%  bottom 15.2%
D039 photo      top 11.9%  bottom 12.3%
D040 photo      top 10.5%  bottom  8.5%
D041 photo      top 11.1%  bottom 16.3%
D042 photo      top 16.6%  bottom 15.1%
D043 photo      top 13.3%  bottom 13.2%
D044 photo      top 13.0%  bottom 13.3%
D045 photo      top 33.2%  bottom  0.1% <- text at the edge
D046 photo no text
D047 photo      top  1.6%  bottom 38.3% <- text at the edge
D048 photo      top 15.1%  bottom 28.0%
D049 photo      top 18.7%  bottom 30.4%
D051 scan_image top 30.8%  bottom 31.0%

Two pictures have text at the edge: D045 at the bottom (0.1%) and D047 at the top (1.6%). They are exactly the two photos whose edge is cut off. Every other photo has a margin of at least 8% above and below. Some scans come close: D020 has 2.4% at the top, because the paper fills its scanner page. So this measurement finds the cut-off top of D047 without the model’s reason, and without the contract. It needs a threshold that you must check on more documents, but it does not depend on what the model says about itself. The project does not include this check; you can add it as a seventh signal.

Knowledge check

Check your understanding

Explain how the pipeline knows that D047's vendor is not 'not printed', although the page reads well and the model is 99% sure.

Show the answer

Not from the picture’s signals: OCR read 255 characters with a mean score of 0.99. And not from the model: it said not_printed with a confidence of 0.99. The pipeline knows from the extraction contract. The contract says that the vendor may be missing only as unreadable, because every receipt prints its shop. So “vendor: absent, not_printed” is a reason that the contract does not allow. field_signals flags it, and the date in the same way. The honest reason is cut_off: the top of the photo lost them. (The contract would flag cut_off for the vendor too, because it allows only unreadable. That rule is too strict for a cut-off top, and a better contract would allow cut_off: an improvement you can make, not done in the course. Either way, the receipt gets a flag.) A person sees the flag and asks the customer for a better photo.

Check your understanding

Predict: D022 at half its size (425 x 550 pixels). Which signal will uncertainty give, and will OCR and the model still read its values?

Show the answer

At half size, D022’s text is about 11 pixels high, under the limit of 14. So the page is flagged low_resolution. Its mean score, 0.93, is above 0.75, so it is not unreadable. OCR still reads 7 of its 8 values, raw and prepared. The model’s recorded answer at half size reads all 8. So the signal is a warning that comes before the failure. At a quarter size, OCR reads 0 of 8, and the model says that the page is unreadable.

Check your understanding

Practical check: run python -m doc_extract.uncertainty D045. For each signal and each absent field, say what an agent should do.

Show the answer

The output flags three things. currency: unverified: the model concluded CAD from the address in Toronto, and no line shows it. The agent confirms it (a shop in Toronto charges in Canadian dollars) or asks. total: absent (cut_off) and payment_method: absent (cut_off): the values exist on the paper, below the edge of the photo. The agent asks the customer for a new photo, or types them from the paper receipt as a correction. The shipping is absent with not_printed: till receipts have no shipping line, so the agent does nothing. No agent should compute the total from the subtotal and the tax.

Module recap

This module used the vision model on purpose. In Input and prompt design, you chose its input and wrote a precise prompt. The page and a crop when the receipt is small; a strict schema where “absent” is a normal answer. Invented values went from 11 to 0. In Combine methods, you read with the rules first and called the model for 9 of 54 receipts. You grounded its values on the page and kept disagreements for a person: 427 of 435 values right, at US$0.0018 per document. In this lesson, you measured what the pictures do not let the pipeline know, and kept each unavailable value explicit, with its reason. Your project matches the snapshot end-of-m04: python -m pytest -q prints 84 passed. Only README.md differs: the snapshot’s version lists this module’s commands. Copy it from the snapshot if you want it.

Module practice: compare the pipeline with your own labels

Extract the fields of synthetic receipts, and compare them with a ground truth that you write by hand. A ground truth is the set of right answers that you check the pipeline against. You label what the picture shows, then compare field by field, and count each outcome.

Guided: the partner shop’s photo D049

1. Draw the photo as a picture:

   python -m doc_extract.show D049

   D049  IMG_5101.jpg  (168,957 bytes, photo)
     page 1: 1200 x 1600 pixels at 100 dpi -> out/show/D049-p1.png

   Open the picture in out/show/.

2. Write your labels before you run anything, one per field, as the contract says. Here is the course author’s labelling of D049, read from the photo:

   vendor          Maple & Pine Hardware   (MAPLE & PINE, a partner store of Larkfield)
   order_number    absent, not_printed     (INV# 0271480 is an invoice number)
   date            2026-09-23              (Sep 23 2026)
   currency        CAD                     (inferred: a shop in Kingston ON; no currency printed)
   subtotal        27.45                   (MERCH)
   shipping        absent, not_printed
   tax             3.57                    (Sales tax 13%)
   total           31.02                   (AMOUNT DUE)
   payment_method  Visa                    (TENDER VISA****4821)

3. Run the pipeline:

   python -m doc_extract.combine D049

   D049  IMG_5101.jpg: model used (required fields missing: subtotal, total; an unknown shop: the rules know only Larkfield's layouts)
     vendor          MAPLE & PINE   from both  found      ok
     order_number    (not_printed)  from none  absent     ok
     date            2026-09-23     from both  found      ok
     currency        CAD            from both  inferred   ok
     subtotal        27.45          from model found      ok
     shipping        (not_printed)  from none  absent     ok
     tax             3.57           from both  found      ok
     total           31.02          from model found      ok
     payment_method  Visa           from model found      ok

4. Compare field by field. Give each field one of five outcomes:

   • correct: the same value, after normalising;
   • correct absent: both absent;
   • missing: you have a value, the pipeline has none;
   • wrong: two different values;
   • invented: the pipeline has a value, you have none.

   Here: 7 correct and 2 correct absent. The vendor MAPLE & PINE normalises to Maple & Pine Hardware.

5. Check your labels against the generator’s truth. A hand label can be wrong too:

   python -m doc_extract.documents D049

   D049  IMG_5101.jpg
     kind receipt, source photo, language en, pages 1, 168,957 bytes
     how it was made: {"tilt": 0.05, "angle": 6.0}
     vendor          Maple & Pine Hardware
     order_number    (not on the document)
     date            2026-09-23
     currency        CAD
     subtotal        27.45
     shipping        (not on the document)
     tax             3.57
     total           31.02
     payment_method  Visa
       item: 1 x Rake, steel @ 27.45 = 27.45

   The author’s labels match the truth. This is how the course’s truth was checked: the generator knows each value by construction, and the course author read every receipt again.

Independent: a scan and a cut-off photo

Do the same for the partner shop’s scan D033 and the photo D047, whose top is cut off. Label both from the pictures first (show D033, show D047). Then run combine on each, compare, and count the five outcomes for each receipt. For D047, decide what you write for the vendor and the date. Is it what you know about Larkfield, or what the picture shows?

Show a worked solution

D033, from the pipeline:

D033  scan-partner-1.jpg: model used (required fields missing: subtotal, total; an unknown shop: the rules know only Larkfield's layouts)
  vendor          MAPLE & PINE   from both  found      ok
  order_number    (not_printed)  from none  absent     ok
  date            2026-07-18     from both  found      ok
  currency        CAD            from both  inferred   ok
  subtotal        8.49           from model found      ok
  shipping        (not_printed)  from none  absent     ok
  tax             1.10           from both  found      ok
  total           9.59           from model found      ok
  payment_method  Debit          from model found      ok

D047, from the pipeline:

D047  IMG_4894.jpg: model used (required fields missing: vendor, date)
  vendor          (not_printed)  from none  absent     ok
  order_number    LK-192531      from both  found      ok
  date            (not_printed)  from none  absent     ok
  currency        CAD            from both  found      ok
  subtotal        187.00         from both  found      ok
  shipping        (not_printed)  from none  absent     ok
  tax             24.31          from both  found      ok
  total           211.31         from both  found      ok
  payment_method  Visa           from both  found      ok

The truth of D047:

D047  IMG_4894.jpg
  kind receipt, source photo, language en, pages 1, 176,420 bytes
  how it was made: {"tilt": 0.04, "angle": 2.0, "crop_top": 0.2864, "layout": "till"}
  vendor          (not on the document)
  order_number    LK-192531
  date            (not on the document)
  currency        CAD
  subtotal        187.00
  shipping        (not on the document)
  tax             24.31
  total           211.31
  payment_method  Visa
    item: 1 x Step ladder @ 79.00 = 79.00
    item: 2 x Garden chair @ 54.00 = 108.00

D033: 7 correct, 2 correct absent (the order number and the shipping). The model gave the subtotal, the total and the payment, all grounded on OCR lines of the scan.

D047: 7 correct and 2 correct absent: the vendor and the date are absent in the truth too. The generator’s truth records what the picture shows, and the top of this photo is cut off. If you labelled the vendor “Larkfield” because you know the layout, your label is not what the picture shows. Your comparison would then count a missing value that no reader could find. Label the picture, not your knowledge. The truth also lists printed_but_cut_off: vendor, date: the reason. So the honest reason for both is cut_off. The contract allows cut_off for the date, but only unreadable for the vendor, because it expects every shop name to be readable. A cut-off top shows that this rule is too strict: a better contract would allow cut_off for the vendor too. That is an improvement you can make; the course keeps the rule. The pipeline says absent for both, but with the model’s wrong reason not_printed, and uncertainty D047 flags exactly that.

Completion check: keep unavailable values explicit

Keep unavailable values explicit rather than inventing plausible text. For each case, say what the pipeline stores, why, and what a plausible invented value would have been:

1. D045: the total.
2. D046: the vendor.
3. D047: the date.

Show a worked solution

What combine stores for D045’s money fields and the payment. In python:

from doc_extract import combine, documents
ex, dec = combine.combine(documents.get("D045"))
for n in ("subtotal", "tax", "total", "payment_method"):
    f = ex.fields[n]
    print(f"{n:<15} value={f.value!r:<9} status={f.status:<7} reason={f.reason!r:<12} from={dec.source[n]:<5} flags={dec.flags.get(n, [])}")

subtotal        value='358.95'  status=found   reason=''           from=both  flags=[]
tax             value='46.66'   status=found   reason=''           from=rules flags=['the model says absent (cut_off)']
total           value=None      status=absent  reason='cut_off'    from=none  flags=[]
payment_method  value=None      status=absent  reason='cut_off'    from=none  flags=[]

1. D045 total: value=None, status=absent, reason='cut_off'. The photo ends below the tax line, and the model said so. The plausible invented value is 405.61, the subtotal plus the tax: right by chance on this paper, and with no evidence. The agent asks for a new photo, or adds the total as a correction.
2. D046 vendor: absent, unreadable, as every field of this photo, and the page signal says unreadable: 0 characters found. The plausible invented value is “Larkfield”, because most receipts are. It may even be true, but no reader saw it. The agent asks for a sharper photo.
3. D047 date: absent, with the model’s reason not_printed, and the flag “the contract allows only: cut_off, unreadable”. The plausible invented value is a date from the ticket, such as the day the customer wrote. The honest reason is cut_off. The project keeps the model’s reason and adds the flag, which shows a person that the stated reason is wrong.

In all three cases, the field is not empty and not guessed. It says “absent”, with a reason that tells the agent what to do next.

Recap

• Signals of visual uncertainty are measured, not asked for. They are: too few characters or a low mean OCR score (unreadable), and small text (low_resolution). Also pages that disagree (conflict), model values not on the page (unverified), and absent values with their reason.
• The model’s account of itself is not enough. It said not_printed at 0.99 for D047’s cut-off top, and at a quarter size it wrote six misreads as found. The contract’s allowed reasons catch the first; resolution signals and rules first protect against the second.
• Small pictures fail for every reader at a quarter size. At half size, almost everything still reads, and the low_resolution signal warns first.
• Keep every unavailable value explicit: null, absent and a reason (not_printed, cut_off, unreadable). Never compute or complete a value that the page does not show.

Next, Module 5 checks the values that the pipeline did find: dates, totals that add up, currencies and order numbers. It also decides which documents a person must review.

Next: Business validation