Combine methods

What you will be able to do

Read each receipt with the cheap, exact readers first, the text layer and the rules. Call the vision model only where they are not enough. Then decide, field by field, which value to keep, and give each model value a place on the page. The result is better than either reader alone, at a sixth of the model’s cost.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• The three paths and the rules that fill the contract (fields.py), from Text and OCR paths.
• The evidence record of a value (file, page, box), from Provenance representation.
• The vision model’s recorded answers and prompt v2, from Input and prompt design.
• Citations that point to a source, from Answer and cite in RAG: Building AI That Uses Your Data, if you took it. This lesson does the same for a value on a page.

What you need on your computer

• Your project at the end of the previous lesson: python -m pytest -q prints 77 passed.
• The labs repository in ~/projects/nextia-learning-labs, up to date.

Tested with Python 3.12.13 and RapidOCR 3.10.0 on macOS (Apple silicon). The Windows and Linux commands are not tested. The partner shop, Maple & Pine Hardware, is made up by the course’s generator, like every document.

Get the lesson’s file

1. In the project folder, copy the module and its test from the snapshot:

   macOS / Linux

   S=../nextia-learning-labs/C13/snapshots/end-of-m04
   cp $S/doc_extract/combine.py doc_extract/
   cp $S/tests/test_combine.py tests/

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C13\snapshots\end-of-m04"
   Copy-Item "$S\doc_extract\combine.py" doc_extract\
   Copy-Item "$S\tests\test_combine.py" tests\

2. Run the tests:

   python -m pytest -q

   The last line is 81 passed. The first run reads a few photos with OCR, so it takes about 15 seconds.

Your own bank, and someone else’s

You read your own bank’s statements in seconds: you know where the balance is. Then a statement from another bank arrives, in another layout. You ask a friend who reads any layout. Your friend says: “The balance is 1,240.50.” A careful person does one more thing before they write it down: they find 1,240.50 on the paper. If it is there, they mark the place. If it is not there, they do not use the number until somebody checks.

And if you and your friend read the same statement and get two different numbers, you do not pick one. You look again, together.

That is this lesson. The rules are you: fast, exact, and good only with layouts they know. The vision model is the friend: slower, paid per page, and able to read layouts it has never seen. The careful step in the middle has a name.

The problem: two readers, each with a gap

The partner shop, Maple & Pine Hardware, prints its receipts in its own words. Compare the paths on its digital receipt D017:

python -m doc_extract.compare D017

D017  maple-pine-0534-560.pdf  (digital_pdf, en)
field           true value    text            ocr             vision
vendor          Maple & Pine HardwareMAPLE & PI ok   MAPLE & PI ok   MAPLE & PI ok
order_number    -             - ok            - ok            - ok
date            2026-09-06    2026-09-06 ok   2026-09-06 ok   2026-09-06 ok
currency        CAD           CAD ok          CAD ok          CAD ok
subtotal        9.99          - WRONG         - WRONG         9.99 ok
shipping        -             - ok            - ok            - ok
tax             1.30          1.30 ok         1.30 ok         1.30 ok
total           11.29         - WRONG         - WRONG         11.29 ok
payment_method  Cash          - WRONG         - WRONG         Cash ok

text    6/9 correct, 0.00 s
ocr     6/9 correct, 0.68 s
vision  9/9 correct, 5.86 s, 1269 + 490 tokens = US$0.0074 (recorded)

The rules know only Larkfield’s words: Subtotal, TOTAL, Payment. This shop prints MERCH, AMOUNT DUE and TENDER. Look at its text layer:

python -m doc_extract.textlayer D017

maple-pine-0534-560.pdf, page 1: 18 lines from the text layer (215 x 273 pt)
  [  68.6   25.3  146.8   36.3]  MAPLE & PINE
  [  55.9   39.7  159.5   47.7]  HARDWARE - Kingston ON
  [  68.8   51.7  146.6   59.7]  Larkfield partner store
  [  11.3   69.3   65.7   77.8]  INV# 0534560
  [ 160.6   69.3  204.1   77.8]  Sep 6 2026
  [  11.3   85.3   91.6   93.8]  QTY DESCRIPTION
  [ 178.1   85.3  204.1   93.8]  PRICE
  [  13.7  101.3   86.0  109.8]  1 Potting soil 30 L
  [ 187.6  101.3  204.1  109.8]  9.99
  [  28.3  113.5   60.7  120.5]  @ 9.99 ea
  [  56.7  130.3   87.9  138.8]  MERCH
  [ 187.6  130.3  204.1  138.8]  9.99
  [  56.7  142.3  111.0  150.8]  Sales tax 13%
  [ 187.6  142.3  204.1  150.8]  1.30
  [  56.7  155.1  124.5  165.1]  AMOUNT DUE
  [ 179.1  155.1  204.1  165.1]  11.29
  [  11.3  172.3   72.3  180.8]  TENDER CASH
  [  45.5  190.7  169.9  196.2]  Made-up document for a course. Not a real receipt.

So the rules miss the subtotal, the total and the payment. The course made this layout on purpose, as the unknown layout that every real pipeline meets one day. The model reads all nine fields right, for US$0.0074.

So why not send every page to the model? Three reasons:

• Cost and time. The model on all 54 receipt pages costs US$0.596, about US$0.011 per document, and about 7 seconds per page. The rules on the text layer take milliseconds.
• No place. The model gives values without a box. An agent cannot check them in a second, as Provenance representation showed.
• Its own mistakes. On the clean PDF D009, the text layer has LARKFIELD and the model writes LARKEFIELD.

Each reader has a gap that the other one fills.

The idea: rules first, the model when needed, then reconcile

The project’s combine.py reads each receipt in three steps.

1. Read and apply the rules. Each page is read the way its route says: the text layer, or OCR on the prepared picture. The rules fill the contract.
2. Call the model only when the rules are not enough. The function why_model gives three reasons. A required field is missing, the shop is unknown, or a required value has a low reading score: the score that OCR gave its line is under 0.80. With no reason, the model is not called.
3. Reconcile, field by field. To reconcile is to compare the two answers and decide what to keep. Both give the same value: keep the rules’ value and its box. They give two values: keep the rules’ value, and mark a conflict for a person. Only the model gives a value: look for it on the page. Only the rules give a value: keep it, with the model’s note. Neither gives a value: keep “absent”, with the model’s reason.

The step “look for it on the page” is grounding: tying a model’s value to a line on the page that shows it. A grounded value gets that line’s box, so it has evidence like any other value. A value that cannot be found on the page is unverified, and a person must check it. This is the careful friend’s habit, written as code.

Routing and reconciliation in combine.py. The model is called for 9 of the 54 receipts. Its values are kept only with a place on the page; a disagreement or a value that cannot be found goes to a person.

Text description of the diagram

A receipt is read page by page by its route, then the rules apply. For 45 of the 54 receipts, the rules are enough. Their values are kept with their boxes, and the model is not called. For the other 9, a required field is missing or the shop is unknown, so the project uses the model’s recorded answer. Then each field is reconciled. If both give the same value, the rules’ value and box are kept. If they give two values, the rules’ value is kept and the field is a conflict for review. If only the model gives a value, the project looks for it on a line of the page. If the line is there, the value gets that line’s box. If not, the value is unverified and goes to review. If only the rules give a value, it is kept with the model’s note. If neither gives a value, the field is absent, with the model’s reason.

A tiny example: grounding on two lines

Grounding compares normalised values, not characters. A photo prints the total as 1 011,78 $, and the model answers 1011.78. Build a page with two OCR lines and ask ground three questions. Start python in the project folder and type:

from doc_extract import combine
from doc_extract.pages import Line, Page
page = Page("x.jpg", 1, 500, 500, "px", "ocr",
            [Line("TOTAL", (0, 0, 50, 10)), Line("1 011,78 $", (300, 0, 380, 10))])
print(combine.ground("total", "1011.78", "", [page]))
print(combine.ground("total", "1101.78", "", [page]))
print(combine.ground("order_number", "LK-123456", "LK-123456", [page]))

Line(text='1 011,78 $', box=(300, 0, 380, 10), score=1.0, size=0.0, bold=False, angle=0.0)
None
None

The first value is on the page: contract.money('1 011,78 $') is 1011.78, the same as the model’s value, so ground returns the line and its box. The second value has two digits swapped: no line shows it, so None. The third value, an order number, is not on the page at all: None again. A None makes the value unverified.

Combine three receipts

1. A Larkfield receipt with a text layer:

   python -m doc_extract.combine D011

   D011  receipt-lk-103119.pdf: model not needed
     vendor          Larkfield      from rules found      ok
     order_number    LK-103119      from rules found      ok
     date            2026-08-22     from rules found      ok
     currency        CAD            from rules found      ok
     subtotal        189.00         from rules found      ok
     shipping        (not_found)    from rules absent     ok
     tax             22.68          from rules found      ok
     total           211.68         from rules found      ok
     payment_method  Cash           from rules found      ok

   The rules found every required field, so the model was not called. The receipt costs nothing, and every value has the rules’ box. The shipping is absent with the reason not_found. That is not one of the contract’s three reasons: the rules write it when no line matches, as in Provenance representation. They know that the value is missing, but not why. Shipping is not required, so it is no reason to ask the model.

2. The partner shop’s receipt:

   python -m doc_extract.combine D017

   D017  maple-pine-0534-560.pdf: model used (required fields missing: subtotal, total; an unknown shop: the rules know only Larkfield's layouts)
     vendor          MAPLE & PINE   from both  found      ok
     order_number    (not_printed)  from none  absent     ok
     date            2026-09-06     from both  found      ok
     currency        CAD            from both  inferred   ok
     subtotal        9.99           from model found      ok
     shipping        (not_printed)  from none  absent     ok
     tax             1.30           from both  found      ok
     total           11.29          from model found      ok
     payment_method  Cash           from model found      ok

   The first line gives both reasons. Read the from column. Four values come from both: the rules and the model agree, so the project keeps the rules’ value and box. Three values come from model: subtotal, total and payment. Each was found on a line of the page, so each is found with a box. Two fields are absent from none, with the model’s reason not_printed: this shop prints no order number and no shipping.

3. A photo whose bottom is cut off:

   python -m doc_extract.combine D045

   D045  IMG_4880.jpg: model used (required fields missing: currency, total)
     vendor          Larkfield      from both  found      ok
     order_number    LK-754892      from both  found      ok
     date            2026-06-13     from both  found      ok
     currency        CAD            from model unverified ok  [unverified: the model's value cannot be found on the page]
     subtotal        358.95         from both  found      ok
     shipping        (not_printed)  from none  absent     ok
     tax             46.66          from rules found      ok  [the model says absent (cut_off)]
     total           (cut_off)      from none  absent     ok
     payment_method  (cut_off)      from none  absent     ok

   This is the careful case. The currency CAD comes from the model, but no line shows “CAD” or a currency sign. The model concluded it from the shop’s address in Toronto. Grounding cannot find it, so its status is unverified, and it goes to a person. The total and the payment stay absent, with the model’s reason cut_off. Nobody invents them. The tax shows the last reconciliation case: the rules read 46.66 on the line at the cut edge, and the model said cut_off. The project keeps the value that has a box, and keeps the model’s note.

The picture shows where the values of two of these receipts come from. On the left is another partner receipt, the photo D049. Its result has the same shape as D017’s; you run it yourself at the end of the lesson.

Where the combined values come from: the rules (green), model values grounded on a line (blue), and one grounding that lands on the wrong line (orange). On D045, the cut edge leaves the total absent and the currency unverified.

Text description of the diagram

On the left is the photo D049 from the partner shop. Green boxes mark the values that the rules read and the model confirmed. They are the shop name, the address, the invoice line with the date, and the tax line. A blue box marks the total, 31.02, found on the AMOUNT DUE row, and another the payment line. An orange box marks the subtotal, 27.45. Grounding found it on the item’s price line, the first line that shows 27.45. It is not on the MERCH line, which also shows 27.45. On the right is the photo D045, cut off at the bottom. Green boxes mark the shop name, the date, the order number, the subtotal and the tax line at the cut edge. A red line marks the edge. The total and payment are absent with the reason cut_off, and the currency has no box because it is not printed.

Look at badge 1. On this receipt, 27.45 is printed three times: the item’s price, the unit price and the MERCH line. Grounding proves that the page shows the value. It does not prove which line the value belongs to: ground takes the first line with the same amount. Here the value is right, and the box is on a line that an agent can still check. On a receipt with two items, the first match could be a different line with the same amount by chance. Keep this limit in mind; the knowledge check comes back to it.

The whole receipt set

The course ran combine on all 54 receipt documents and counted the results:

54 receipt documents; model used on 9: D017, D018, D033, D034, D045, D046, D047, D048, D049
reasons: {'required fields missing': 9, 'an unknown shop': 6}
model cost: US$0.097 in all, US$0.0018 per document
field values by source: {'rules': 406, 'both': 34, 'none': 27, 'model': 19}
flags: {'unverified': 1, 'the model says absent': 1}
printed values right: 427 of 435 (98.2%)

The model was called for 9 documents: the 6 receipts of the partner shop, and the three damaged photos D045, D046 and D047. The other 45 documents cost nothing beyond the rules. The model’s 9 calls cost US$0.097 in all: US$0.0018 per document, about a sixth of the model on every page.

The quality is the best of all the paths:

• One reader alone
• Model alone, prompt v2
• Rules first, model when needed

Printed values right on the 54 receipt documents (435 values). The text layer reads only digital PDFs; OCR here reads the prepared picture. Combining the readers beats each one alone.

Text description of the chart

Printed values right, of 435. Text layer and rules: 38.9%. OCR on the prepared picture and rules: 93.8%. The vision model with prompt v2: 97.7%. Rules first and the model when needed: 98.2%.

Combined, 427 of the 435 printed values are right (98.2%), with 0 wrong and 0 invented. The 8 missing values are all on D046, the photo that nobody can read. 53 of the 54 receipts are complete. The model alone got 425 right, and its mistakes included D009’s vendor. In the combined pipeline, the rules read D009, so the misread is never used.

Mei sums it up for Omar: “The rules do most of the work for nothing. The model fills the gaps of the rules, and the rules check the model, because every model value needs a line on the page.”

When to combine, and what it costs

• Combine when you have an exact reader for most documents and a gap. Here the text layer and the rules read Larkfield’s receipts, and the gap is another shop and damaged photos. The model’s cost then follows the size of the gap: 9 calls for 54 receipts.
• Do not combine when no exact reader exists. For a new kind of document with no rules, the model reads everything, and you still ground its values.
• What it costs: two readers to maintain, and reconciliation code with its own tests. The rules must report what they could not find, or why_model cannot see a gap.
• Grounding has limits. It needs the OCR lines or the text layer of the page. It cannot ground an inferred value, such as D045’s currency. And when a value is printed more than once, it takes the first line that shows it.

Common mistake

The model's answer wins

Symptom: A teammate simplifies the pipeline: “Send every page to the model. When the rules and the model disagree, trust the model: it reads better.” The agents start to see LARKEFIELD as the vendor of a Larkfield receipt.

Cause: The course ran that design on all 54 receipts. It listed every field where both readers give a value and the values differ:

D009 vendor | rules Larkfield right | model LARKEFIELD wrong
model on all 54 documents: US$0.596, US$0.0110 per document

On this set, the two readers disagree on one field only: D009’s vendor. The rules are right, and the model is wrong. The simplified design keeps the wrong value and drops the box of the text layer. And it costs six times as much: US$0.596 instead of US$0.097 for the 54 documents.

Diagnose: Count the disagreements and check them against the truth before you choose a side. Here there is one, and the model is the reader that is wrong.

Fix: Keep the rules first and call the model only for a reason. When both give a value and they differ, do not choose. Keep the value with evidence, and send a conflict to a person, as combine.py does. The conflict path did not fire on this set, because the model was not called for D009. A test with two different values checks that it works.

Consequence in AI work: a disagreement between two readers is information. A pipeline that hides it, either way, throws away its best signal of an error.

Your turn: the partner shop’s photo

Change one condition: the partner shop’s photo D049 instead of its digital receipt D017. The photo is read by OCR, not the text layer. Predict which fields come from the model, and on which line the subtotal 27.45 is grounded. Then run:

python -m doc_extract.combine D049

And print the lines behind each value. In python:

from doc_extract import combine, documents
ex, dec = combine.combine(documents.get("D049"))
for name, f in ex.fields.items():
    print(f"{name:<15} {dec.source[name]:<5}", [(ln.text, [round(v) for v in ln.box]) for ln in f.lines])

Show the answer

D049  IMG_5101.jpg: model used (required fields missing: subtotal, total; an unknown shop: the rules know only Larkfield's layouts)
  vendor          MAPLE & PINE   from both  found      ok
  order_number    (not_printed)  from none  absent     ok
  date            2026-09-23     from both  found      ok
  currency        CAD            from both  inferred   ok
  subtotal        27.45          from model found      ok
  shipping        (not_printed)  from none  absent     ok
  tax             3.57           from both  found      ok
  total           31.02          from model found      ok
  payment_method  Visa           from model found      ok

vendor          both  [('MAPLE & PINE', [358, 300, 734, 386])]
order_number    none  []
date            both  [('INV# 0271480', [116, 537, 383, 606]), ('Sep 23 2026', [780, 475, 1018, 546])]
currency        both  [('HARDWARE - Kingston ON', [305, 361, 800, 452])]
subtotal        model [('27.45', [913, 616, 1032, 675])]
shipping        none  []
tax             both  [('Sales tax 13%', [366, 841, 627, 909]), ('3.57', [952, 796, 1048, 854])]
total           model [('31.02', [920, 853, 1054, 916])]
payment_method  model [('TENDER VISA****4821', [176, 971, 602, 1060])]

The same three fields as on D017 come from the model: subtotal, total and payment. All three are grounded on OCR lines, with boxes in pixels of the photo. The total is on the AMOUNT DUE row (31.02 at y = 853). The subtotal is on the line 27.45 at y = 616: the item’s price, not the MERCH row at y = 745. This is badge 1 of the picture: right value, first matching line.

See it end to endRead real receiptsCase study

The receipt case study combines rules for rupiah with the recorded vision model on real receipt photos. The model is asked for 21 of the 60 test receipts, and the combined path reads 169 of 186 labelled values, for about US$0.004 per receipt.

Knowledge check

Check your understanding

Explain why combine.py called the model for D045, and why its currency CAD is 'unverified' although it is right.

Show the answer

The rules did not find two required fields on the photo: the currency and the total. So why_model gave the reason “required fields missing: currency, total”, and the project used the model’s recorded answer. The model said the total is cut off: it stays absent. The model gave the currency CAD, with the status inferred, from the shop’s address in Toronto. No line of the photo shows “CAD” or a currency sign, so ground found no line. A value without a line has no evidence, so the project marks it unverified and sends it to a person. Right or not, it has nothing on the page that an agent could check.

Check your understanding

Predict: on a receipt with two items, 'Hose 19.95' and 'Gloves 19.95', the rules miss the subtotal and the model answers 39.90. Where does ground put the box? And if the model answered 19.95 by mistake?

Show the answer

For 39.90, ground looks for a line whose amount is 39.90. If the receipt prints a subtotal line with 39.90, the box goes there, and the value is grounded on the right line. For a wrong answer of 19.95, ground finds the first item line with 19.95 and gives it a box. The wrong value is then “grounded”, because it is printed, only on another line. Grounding proves that the page shows a value, not that the value belongs to the field. This is why the next module adds checks across fields: here, the subtotal must equal the sum of the item lines.

Check your understanding

Practical check: run python -m doc_extract.combine D018 and python -m doc_extract.vision D018. Which fields come from the model, and what did the call cost?

Show the answer

The model was used, for both reasons: the subtotal and the total are missing, and the shop is unknown. Three values come from model: the subtotal 47.25, the total 53.39 and the payment Visa. All three are found, so each was grounded on a line of the PDF. Four come from both, and the order number and the shipping are absent from none, with the reason not_printed. The vision D018 line gives the call: 1,269 tokens in and 488 out. At the list price: 1,269 × 2 / 1,000,000 + 488 × 10 / 1,000,000 = US$0.0025 + US$0.0049 = about US$0.0074. The other 45 receipts cost nothing.

Recap

• Rules first: the text layer and the rules read the layouts they know, for no cost, with a box for every value.
• The model when needed: why_model calls it only for a missing required field, an unknown shop or a low reading score. That was 9 of 54 receipts, US$0.0018 per document instead of US$0.0113 per page.
• Reconcile field by field. The same value: keep the rules’ value and box. Two values: a conflict for a person. The model alone: grounded on a line, or unverified. Both absent: absent, with the model’s reason.
• Combined: 427 of 435 printed values right, 0 wrong, 0 invented, 53 of 54 receipts complete.
• Grounding proves that the page shows a value, not which line it belongs to. Trusting the model when the readers disagree would have kept D009’s misread vendor.

Next, you plan for the pictures that nobody can fully read: blurred, cut off, too small, or with pages that disagree.

Next: Handle visual uncertainty