Input and prompt design

What you will be able to do

Decide exactly what a vision model receives: the page, the part of the picture, the fields and their formats. Also tell it what to write when a value is not there. A plain request makes the model fill an empty field with a plausible number. A precise request makes “absent” a normal answer, and a crop makes many pages cheaper.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• The vision path, tokens and the cost of one page, from Text and OCR paths.
• The extraction contract and its three statuses (found, inferred, absent with a reason), from Define the extraction contract.
• A JSON schema for a model’s answer, from Define a schema in Building Reliable Applications with LLM APIs.

What you need on your computer

• Your project at the end of Module 3: python -m pytest -q prints 77 passed. If your project is broken, copy the snapshot end-of-m03, as the snapshots README says.
• The labs repository in ~/projects/nextia-learning-labs, up to date.
• No key and no account. Every model answer in this lesson is a recording that the project replays.

Tested with Python 3.12.13 on macOS (Apple silicon). The Windows and Linux commands are not tested. Every document in this course is made by the course’s generator; no real customer or shop is in it.

Get the lesson’s files

The course recorded more model answers for this module. They are the scans cropped, six pictures at smaller sizes, and every page a second time. They are data files, not code.

1. In the project folder, copy the four recordings and the design notes from the snapshot:

   macOS / Linux

   S=../nextia-learning-labs/C13/snapshots/end-of-m04
   cp $S/recordings/vision-v2-crop.jsonl $S/recordings/vision-v2-half.jsonl recordings/
   cp $S/recordings/vision-v2-quarter.jsonl $S/recordings/vision-v2-pass2.jsonl recordings/
   cp $S/docs/vision-prompt.md docs/

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C13\snapshots\end-of-m04"
   foreach ($f in 'crop', 'half', 'quarter', 'pass2') { Copy-Item "$S\recordings\vision-v2-$f.jsonl" recordings\ }
   Copy-Item "$S\docs\vision-prompt.md" docs\

2. Run the tests:

   python -m pytest -q

   The last line is still 77 passed. This lesson adds data, not code.

A form for a new colleague

Imagine that you hire a temporary colleague to type receipts into a form. You say: “Write down the order number.” On a receipt from the shop’s till, there is no order number. There is a transaction number. Your colleague wants to help, and an empty box looks like a failure. So they copy the transaction number. Nobody told them that an empty box is a correct answer.

Now give them a better form. Each box has a name and a format. A note says: “A transaction number is not an order number. If a value is not printed, write not printed. Never guess.” You also cut the receipt out of the big scanner page, so they do not search a white page for it. The colleague is the same person. The request changed, and so did the result.

A vision model is that colleague. It reads what you send and does what you ask. So you design two things: the input (the picture) and the prompt (the request and the shape of the answer).

The problem: a plain request invents values

The course’s first prompt, v1, was a plain request: “Extract the data from this receipt as JSON with the keys vendor, order_number, date…”. Look at its recorded answer for D003, an in-store receipt from the till:

python -m doc_extract.vision D003 --prompt v1

recorded answer (replayed; nothing was sent)
D003 receipt-t0693-220.pdf page 1, prompt v1: gpt-6.1-sol-2026-09-29, 469 tokens in, 220 out, 4.951 s, about US$0.0031
  vendor          LARKFIELD Home & Garden
  order_number    0693-220
  date            2026-09-23
  currency        CAD
  subtotal        406.0
  shipping        absent (not_given)
  tax             52.78
  total           458.78
  payment_method  Debit ****3310
    item: 1 x Tool chest @ 119.0 = 119.0
    item: 1 x Patio parasol @ 129.0 = 129.0
    item: 2 x Step ladder @ 79.0 = 158.0

The order number is 0693-220. Now read the top of the receipt in its text layer, each line with its box in points:

python -m doc_extract.textlayer D003

receipt-t0693-220.pdf, page 1: 31 lines from the text layer (227 x 351 pt)
  [  75.6   22.7  151.2   36.7]  LARKFIELD
  [  80.2   39.7  146.5   48.2]  Home & Garden
  [  -3.9   52.2  230.7   60.7]  larkfield.example · help desk 1-800-555-0199
  [  14.2   64.3  208.6   73.3]  ------------------------------------
  [  14.2   76.8  100.6   85.8]  Date: 2026-09-23
  [ 185.6   76.8  212.6   85.8]  20:43
  [  14.2   89.3  127.6   98.3]  Transaction: 0693-220
  [  14.2  101.8   52.0  110.8]  Till 03
  [  14.2  114.3  208.6  123.3]  ------------------------------------

There is no order line. 0693-220 is the till’s transaction number. The model did what the temporary colleague did: it filled an empty box with the nearest number. This is an invented value: a value for a field that the document does not show.

The course checked all 54 receipt pages in the same way. It listed every field that a receipt does not print, but that the model filled anyway. Next to each value is the printed line that it came from:

D003 digital_pdf order_number 0693-220 | printed: Transaction: 0693-220
D007 digital_pdf order_number 0530-850 | printed: Transaction: 0530-850
D013 digital_pdf order_number 0838-785 | printed: Transaction: 0838-785
D017 digital_pdf order_number 0534560 | printed: INV# 0534560 Sep 6 2026
D018 digital_pdf order_number 0727251 | printed: INV# 0727251 Jul 13 2026
D028 scan_image order_number 0917-319 | printed: Transaction: 0917-319
D033 scan_image order_number 0587497 | printed: INV# 0587497 Jul 18 2026
D034 scan_image order_number 0866699 | printed: INV# 0866699 Jul 15 2026
D042 photo order_number 0272-973 | printed: Transaction: 0272-973
D048 photo order_number 0631658 | printed: INV# 0631658 Jun 6 2026
D049 photo order_number 0271480 | printed: INV# 0271480 Sep 23 2026
invented values: 11

Eleven invented values, all order numbers. Six are Larkfield transaction numbers. Five are invoice numbers of the partner shop, Maple & Pine Hardware. Each one is a number that is really printed, so it looks right to a person who reads fast. An agent who searches the order system for 0693-220 finds nothing, or worse, a different order.

The overall score hides this problem. Prompt v1 gets 424 of the 435 printed values right (97.5%). But only 40 of the 54 receipts are complete, with every required value right and nothing invented.

The same page, two prompts. The plain request copies the transaction number into the order field. The precise request answers absent, with the reason not_printed.

Text description of the diagram

On the left is the top of receipt D003. It shows the shop name, the date and time, a transaction number in an orange box, and the till number. It has no order line. On the right, the answer to prompt v1 gives the order number 0693-220, which is the transaction number. The answer to prompt v2 gives the order number as absent, with the reason not_printed. Its rules say that a transaction number is not an order number, and that the model must never guess.

The idea: design the input and the prompt

A vision model answers the request that you send. So the request is a design, with two parts.

Input design is the choice of picture. Send the page that holds the receipt, not every page of the file. Send only the part of the page that is the document, when the document is small on the page. And send the picture at a size where the text is readable. In this project, PDF pages are drawn at 150 dpi, and every picture is at most 1,600 pixels on its longer side.

Prompt design is the request and the shape of the answer. Prompt v2 has three parts:

• A strict JSON schema for the answer. A schema here is the extraction contract, written for the model. Each field is an object with five keys: value (normalised), printed (as on the page), status (found, inferred or absent), reason, and confidence.
• Format rules: dates as YYYY-MM-DD, and money with two decimals and a point. Dates on receipts from Quebec have the day first. Tax is the sum of all tax lines.
• Rules for absent values: “A transaction or invoice number is NOT an order number.” And: “If a value is not printed, is cut off, or you cannot read it, set value and printed to null, status absent, and reason not_printed, cut_off or unreadable. Never guess, compute or complete a value.”

The last rule is the note on the colleague’s form. It makes “absent” an answer that the schema expects, with a place to say why. To see one field of the schema, start python in the project folder and type:

import json
from doc_extract import vision
s = vision.schema_v2()
print(s["required"])
print(json.dumps(s["properties"]["fields"]["properties"]["total"]))

['readable', 'fields', 'items']
{"type": "object", "additionalProperties": false, "properties": {"value": {"type": ["string", "null"]}, "printed": {"type": ["string", "null"]}, "status": {"type": "string", "enum": ["found", "inferred", "absent"]}, "reason": {"type": ["string", "null"]}, "confidence": {"type": "number"}}, "required": ["value", "printed", "status", "reason", "confidence"]}

value may be null: the schema itself allows an empty box. status can take only three words. In strict mode, the model’s answer must match this shape exactly, so the project never parses free text. The top level also has readable: the model can say that the whole page cannot be read.

Prompt v1 has 32 words. Prompt v2 has 218 words, plus the schema. You can read both in doc_extract/vision.py, and docs/vision-prompt.md compares them side by side.

A tiny example: the same page, prompt v2

Run the recorded answer of prompt v2 for D003:

python -m doc_extract.vision D003

recorded answer (replayed; nothing was sent)
D003 receipt-t0693-220.pdf page 1, prompt v2: gpt-6.1-sol-2026-09-29, 1359 tokens in, 327 out, 5.102 s, about US$0.0060
  vendor          LARKFIELD (confidence 1)
  order_number    absent (not_printed) (confidence 1)
  date            2026-09-23 (confidence 1)
  currency        CAD (confidence 1)
  subtotal        406.00 (confidence 1)
  shipping        absent (not_printed) (confidence 1)
  tax             52.78 (confidence 1)
  total           458.78 (confidence 1)
  payment_method  Debit (confidence 1)
    item: 1 x Tool chest @ 119.00 = 119.00
    item: 1 x Patio parasol @ 129.00 = 129.00
    item: 2 x Step ladder @ 79.00 = 158.00

Compare the two answers, field by field:

Field	Prompt v1	Prompt v2	What made the difference
order_number	0693-220	absent (not_printed)	the rule about transaction numbers, and “never guess”
subtotal	406.0	406.00	the format rule: two decimals
payment_method	Debit ****3310	Debit	the rule: no card number
shipping	absent (not_given)	absent (not_printed)	a reason from the contract’s list

The precise prompt costs more. Follow the numbers on the two cost lines. The list price is US$2 per million tokens in and US$10 per million tokens out:

• v1: 469 × 2 / 1,000,000 + 220 × 10 / 1,000,000 = US$0.0009 + US$0.0022 = about US$0.0031.
• v2: 1,359 × 2 / 1,000,000 + 327 × 10 / 1,000,000 = US$0.0027 + US$0.0033 = about US$0.0060.

The 890 extra tokens in are the longer prompt and the schema. The extra tokens out are the printed, status, reason and confidence of every field. Over all 54 pages, the median page costs US$0.0060 with v1 and US$0.0113 with v2. Prices change: check your provider’s current price page.

This is what the extra cost buys, on the same 54 pages:

	Prompt v1	Prompt v2
Printed values right	424 of 435 (97.5%)	425 of 435 (97.7%)
Invented values	11	0
Complete receipts	40 of 54	51 of 54

The share of right values hardly moves. The invented values go to zero, and eleven more receipts are complete. The invented values were the expensive mistakes: plausible, and wrong in the field that agents search by.

Replay: the answers are recordings

Every answer in this lesson is a recording. The course sent each page to the model and saved the answer, the model name, the tokens and the seconds in recordings/vision-*.jsonl. vision.py replays the saved answer by default, so you need no key and spend no money. The diagram shows how one request is made, and the two ways back to the pipeline:

One request to the vision model: the page, the picture, the prompt and the schema. The course saved one real answer per page; the project replays it unless you call a model yourself with --live.

Text description of the diagram

The request starts from the document. The project chooses the receipt page only and draws it at 150 dots per inch. It crops the picture to the receipt if you ask, and makes sure that it is at most 1,600 pixels long. The picture, prompt v2 and the JSON schema make one request. With the option –live, the project sends it to a vision model with your own key, and gets back JSON text, tokens and seconds. The course did this for every page and saved the answers in the recordings folder. By default, the project replays the saved answer, with no key and no cost. Either way, to_extraction turns the answer into the same record that the text layer and OCR give.

The live call is optional. Without a key, it stops before it sends anything:

python -m doc_extract.vision D036 --live

A live call needs VISION_BASE_URL, VISION_API_KEY and VISION_MODEL (see .env.example).

You can set these three values in .env for your own OpenAI-compatible endpoint. Then the same command sends the page and prints a live answer. At the list price above, it costs about one US cent per page. Your answers can differ a little from the recordings: another model, another day.

How much do the answers change? The course sent every receipt page twice with prompt v2 (vision-v2-pass2.jsonl). Then it compared the two answers, field by field:

D028 vendor 'LARKFIELD' -> 'LARKFIELD Home & Garden'
486 values; changed after normalising: 0; written differently: 1

In this run, no normalised value changed, and the model wrote one vendor name in a longer form. That is good news, but it is one measurement of one model. A recording also has a second use: months later, an auditor can see exactly what the model said.

Send only the part that matters

D019 is a scanned receipt: a narrow strip of paper on a letter-size scanner page. The model reads the whole picture, white paper included, and the size of the picture decides most of the tokens in. Compare the sizes of the two pictures that the project can send. In python:

from doc_extract import documents, vision
d = documents.get("D019")
for crop in (False, True):
    jpeg, size = vision.picture(d.path, 1, crop)
    print("crop" if crop else "full", size)

full (1236, 1600)
crop (440, 624)

The crop uses prepare.content_box, the function that crops pictures before OCR in Image preparation. Now compare the two recorded answers:

1. Show the answer for the whole page:

   python -m doc_extract.vision D019

   D019 scan0001.pdf page 1, prompt v2: gpt-6.1-sol-2026-09-29, 3194 tokens in, 622 out, 8.579 s, about US$0.0126

2. Show the answer for the crop:

   python -m doc_extract.vision D019 --crop

   D019 scan0001.pdf page 1, prompt v2: gpt-6.1-sol-2026-09-29, 1281 tokens in, 686 out, 9.723 s, about US$0.0094

   The rest of the two answers is the same: nine values, all right.

Follow the cost. Whole page: 3,194 × 2 / 1,000,000 + 622 × 10 / 1,000,000 = US$0.0064 + US$0.0062 = US$0.0126. Crop: 1,281 × 2 / 1,000,000 + 686 × 10 / 1,000,000 = US$0.0026 + US$0.0069 = US$0.0094. The tokens in fell by 60%, but the cost fell by only 25%. The answer is a little longer than before, and an output token costs five times as much as an input token.

The two pictures for D019, at the same scale. The crop keeps the receipt and drops the empty scanner page: 60% fewer tokens in, 25% less cost, the same nine values.

Text description of the diagram

On the left is the whole scanned page of D019, 1,236 by 1,600 pixels. It is mostly white, with a narrow receipt in the middle, framed in blue. The model read 3,194 tokens and wrote 622, for US$0.0126. On the right is the crop, drawn at the same scale: only the receipt, 440 by 624 pixels. The model read 1,281 tokens and wrote 686, for US$0.0094.

The course recorded the crop on all 17 scanned receipts. The chart puts them in two groups. In the first group, the receipt is small on the scanner page, and the crop is less than half of the picture. In the second group, the paper fills most of the page.

• Whole page
• Crop

Median tokens in per page, for the whole page and the crop, on the 17 scanned receipts. A crop saves tokens only when the receipt is small on the scanner page.

Text description of the chart

Median tokens in per page. A small receipt on a scanner page, 10 scans: 3,194 for the whole page, 1,273 for the crop. The paper fills most of the page, 7 scans: 3,194 for the whole page, 3,206 for the crop.

Over all 17 scans, the crop takes the median page from 3,194 to 1,297 tokens in, and from US$0.0120 to US$0.0094. The values right go from 137 of 137 to 136 of 137. The failure case below shows the one lost value.

Omar, the product manager, asks the cost question: “So we crop everything?” Mei answers: “We crop when the document is small on the page. That is cheap to know, because the crop box is already computed for OCR. On a photo where the paper fills the frame, a crop changes almost nothing.”

Choose the page

The warranty claim D058 has two pages: a digital form and a scanned receipt. Only page 2 has receipt fields. So vision.py sends receipt_page only, and one claim costs one picture, not two. In your own pipeline, the route of each page decides this, as in Document types. The text layer reads page 1 for free.

When to use these designs, and what they cost

• The precise prompt and schema: always. Per page, they add about 900 tokens in and 370 tokens out: about twice the price of the plain prompt. They removed every invented value in this course’s set. They do not make the model perfect: prompt v2 still reads “LARKEFIELD” on the clean PDF D009.
• A crop: when the document is small on the picture. On 10 of the 17 scans, it saved about 60% of the tokens in, and about a quarter of the cost. When the paper fills the frame, it saves nothing.
• Not too small. A smaller picture costs fewer tokens, until the model cannot read it. Handle visual uncertainty measures where that happens.
• Recordings: for teaching, tests and audit. In production, store each answer in the same way: the model name, the tokens, the seconds and the answer. Then you can explain a value later, and compare a new model on the same pages.

The confidence key stays in the schema because the course measures it. Do not use it to decide anything yet. Confidence and review routing shows what it is worth.

Common mistake

The crop saves money and loses a letter

Symptom: D025 is a scanned receipt that lies on its side, turned 90°. With the crop, the model writes the vendor as LARKEFIELD:

python -m doc_extract.vision D025 --crop

D025 scan0007.pdf page 1, prompt v2: gpt-6.1-sol-2026-09-29, 1281 tokens in, 712 out, 9.613 s, about US$0.0097
  vendor          LARKEFIELD (confidence 0.99)

The answer for the whole page had the right name. The course compared the two recorded answers: the size of each picture, and the vendor that the model wrote.

full [1600, 1236] vendor LARKFIELD Home & Garden | printed 'LARKFIELD\nHome & Garden'
crop [620, 444] vendor LARKEFIELD | printed 'LARKEFIELD'

Cause: Not the resolution. Both pictures come from the page drawn at 150 dpi, so the letters have about the same size in pixels. The crop only removes empty paper. The receipt lies on its side in both. The model read the same letters differently with less paper around them. A model’s reading can change with its input in ways that you cannot predict from the picture. Its confidence, 0.99, gives no warning.

Diagnose: Measure a change of input on many pages, not on one. Compare the values with the truth, or with the text layer or OCR where they read well. This is how the course found the one lost value in 137.

Fix: Keep the readers that do not depend on the model in front of it. In the next lesson, the rules read this scan first, and the model is not even called. And check model values against what you know: “LARKEFIELD” does not match the contract’s vendor list (contract.VENDORS), so contract.normalise cannot turn it into “Larkfield”. The course’s validation does not check the vendor list. Confidence and review routing shows what that costs on D009. The course did not record a picture that was turned upright and then cropped, so it cannot say whether turning first helps.

Consequence in AI work: every saving on the input is a change of the input. Measure it on the same pages, with the same scores, before you adopt it.

Your turn: a scan that fills the page

Change one condition: use the scan D020 instead of D019. Its paper fills almost the whole scanner page. Predict the tokens in and the cost of the crop, compared with the whole page. Then run both commands:

python -m doc_extract.vision D020
python -m doc_extract.vision D020 --crop

Show the answer

D020 scan0002.jpg page 1, prompt v2: gpt-6.1-sol-2026-09-29, 3194 tokens in, 561 out, 7.094 s, about US$0.0120
D020 scan0002.jpg page 1, prompt v2: gpt-6.1-sol-2026-09-29, 3206 tokens in, 561 out, 7.04 s, about US$0.0120

The crop costs the same, and it even has 12 tokens more. The crop box is 1,275 × 1,560 pixels: almost the whole scan. The whole page is made smaller to fit in 1,600 pixels (1,236 × 1,600). The crop is not made smaller, so it is a little larger. Both answers give all nine values right. A crop helps only when the document is small on the picture.

Knowledge check

Check your understanding

Explain to Tomás why prompt v1 wrote 0693-220 as the order number of D003, and which parts of prompt v2 stopped it.

Show the answer

D003 is an in-store receipt. It prints a transaction number, 0693-220, and no order number. Prompt v1 asked for an order_number key and said nothing about values that are not there. So the model filled the box with the nearest number, as a helpful colleague would. Prompt v2 changed three things. Its schema allows null, with the status absent and a reason. A rule says that a transaction or invoice number is not an order number. And a rule says “never guess, compute or complete a value”. Together, they make “absent, not_printed” a normal answer. On the 54 receipt pages, the invented values went from 11 to 0.

Check your understanding

Predict: you crop the phone photo D036 before you send it. The paper fills most of the photo. Will the crop save as much as it did on D019?

Show the answer

No, much less. On D019, the receipt was a narrow strip on a letter page. So the crop removed most of the picture, and 60% of the tokens in. On a phone photo, the paper fills most of the frame. Before OCR, the project crops D036 to the box [144, 32, 1088, 1548]: 944 × 1,516 pixels of a 1,200 × 1,600 photo. The picture stays large, so the tokens in stay close to those of the whole photo, and the answer is as long as before. The course did not record crops of photos, so this is a prediction from the scans, not a measurement.

Check your understanding

Practical check: run python -m doc_extract.vision D022 and python -m doc_extract.vision D022 --crop. The crop has fewer tokens in. Why does it cost more?

Show the answer

The whole page is 2,079 tokens in and 297 out: 2,079 × 2 / 1,000,000 + 297 × 10 / 1,000,000 = US$0.0042 + US$0.0030 = about US$0.0071. The crop is 1,101 tokens in and 558 out: US$0.0022 + US$0.0056 = about US$0.0078. The crop saved 978 tokens in, but the answer was 261 tokens longer. An output token costs five times as much, so the longer answer cost more than the crop saved. The length of an answer varies from call to call. So judge a crop on many pages, not on one.

Recap

• A vision model answers the request that you send. Input design chooses the page, the part of the picture and its size. Prompt design gives a strict schema, format rules and rules for absent values.
• A plain prompt invented 11 order numbers: transaction and invoice numbers, copied. The precise prompt v2 invented none, at about twice the price per page.
• In the schema, value may be null, status is found, inferred or absent, and an absent value has a reason. “Absent” is a normal answer.
• Crop when the document is small on the picture: on 10 of 17 scans, about 60% fewer tokens in and a quarter less cost. Measure every change of input on many pages: the crop of D025 lost one letter of the vendor.
• The project replays recorded answers, with no key and no cost. --live calls a model with your own key.

Next, you call the model only where the rules cannot read a receipt. And you find a place on the page for each of its values.

Next: Combine methods