Read real receipts

What you will be able to do

Take the course’s doc-extract pipeline out of Larkfield, and point it at 100 real receipt photos from Indonesia. Their layout and language are new to its rules. Measure it field by field against the dataset’s own labels. Adapt it one change at a time, on receipts that you keep apart from the test. Then report what works, what fails and why, and what it costs.

Case studyPortfolio piece

Dataset

CORD v2 (Consolidated Receipt Dataset), test split: 100 Indonesian receipt photos with labels, NAVER CLOVA

Licence

CC BY 4.0

Time

About 2.5 hours

Runs in

VS Code, Python 3.12+

Skills used

• Map a dataset’s labels to a contract
• Dev and test receipts
• One change at a time
• Money and words of another country
• OCR on raw or prepared photos
• The model where the rules stop
• Field-level scores and review routing
• Failure analysis

Download the starter project (.zip)Finished project (.zip)

Unzip the starter, open the folder in VS Code, and follow the steps below. They start with a new virtual environment.

The project is the course’s doc-extract at the end of Module 6, unchanged, plus a folder cord_eval. It holds this case study’s code: the sample, the label mapping, the rules for Indonesian receipts, the runs, the scorer and an evaluation notebook. In the starter, the rules know the course’s words only; you add the Indonesian ones. The finished project is the result, for when you are stuck. The code is under the MIT licence.

Before you startWhat to know and to have readyAbout 150 minutes

What you need to know

This case study uses the whole course, on receipts that are not Larkfield’s. These lessons matter most:

• The three paths, and the rules that fill the contract, from Text and OCR paths.
• Crop, turn and straighten before OCR, from Image preparation.
• Rules first, the model when needed, and grounding, from Combine methods.
• The checks that a total adds up, from Business validation.
• Outcomes per field (correct, missing, wrong), from Field-level metrics.

What you need on your computer

• Python 3.12 or newer, and VS Code with its Python and Jupyter extensions.
• About 600 MB of free disk space: the packages (the course’s own, plus matplotlib) and the sample (19.7 MB).
• About 1.5 GB of free memory while OCR runs. OCR of the 100 photos takes a minute or two, once; later runs read a cache.

Tested on macOS (Apple M4, 16 GB, Python 3.12.13, RapidOCR 3.10.0). The Windows and Linux commands are written, not tested. Every time is from the author’s computer while other work ran on it: your numbers will differ.

The situation

Dewi Lestari runs finance operations for a company with offices in Jakarta and Toronto. Staff claim their meals and taxis with a photo of the paper receipt. In Jakarta, two people type the totals of about 400 receipts a week. The Toronto team already reads its receipts with a pipeline that one of its engineers built while taking this course. The people are invented; the receipts are real.

Dewi asks Arif Nugroho, a data engineer: “Can we use Toronto’s pipeline for our receipts? I want three answers, with numbers. How many totals does it read right? Which receipts must a person still check? And what does it cost?”

The pipeline reads Canadian receipts in English and French. Sam, an intern, asks the first question: “Why not just try it on our receipts and see?” Arif agrees, with one condition: “We need receipts where we know the right answer. Otherwise ‘see’ means ‘guess’.”

The objective. On photos of Indonesian receipts, read the total, and the subtotal and tax when the receipt prints them. Keep the place of each value on the photo. Send to a person every receipt that the pipeline is not sure about. Measure it on receipts that nobody used to build it.

What you will build

The case study in one picture: adapt on the dev receipts, freeze, then measure the whole path once on the test receipts.

Text description of the diagram

The 100 receipts split into 40 dev receipts and 60 test receipts. On the dev receipts, the rules are adapted one change at a time, then frozen. On each test receipt, the rupiah rules read the raw photo. If they find a total that adds up, their values are kept with their boxes. If not, the recorded vision model reads the receipt, 21 times in all. Each field is then reconciled: agreeing or grounded values are kept, conflicts and unverified values go to review. A receipt with no total, or whose amounts do not add up, also goes to review. The rest go straight through.

Part	What you do	Measured on
1	Run the pipeline as it is: the baseline	dev receipts
2	Read rupiah: a new money reader	dev receipts
3	Add the Indonesian words; compare raw and prepared photos	dev receipts
4	Call the model where the rules stop	recorded answers
5	The fair comparison, once	test receipts
6	One receipt, end to end	a test receipt

The data

The receipts are CORD v2, the Consolidated Receipt Dataset by NAVER CLOVA. It has 1,000 photos of receipts from shops, restaurants and cafés in Indonesia, with labels made by people. This case study uses the 100 receipts of its test split.

Licence and attribution. CC BY 4.0, stated by the creators on GitHub and on the dataset card. Credit: “CORD v2 by NAVER CLOVA, CC BY 4.0”. Cite the paper by Park and others, CORD: A Consolidated Receipt Dataset for Post-OCR Parsing (2019). CC BY lets us share a copy if we say what we changed. The original split is one file of 234 MB. So the course keeps a fixed sample of 19.7 MB in the labs repository. It holds the 100 pictures as JPEG, at most 2,000 pixels long, and the labels unchanged. The dataset card lists the changes and the checksums.

Arif compared CORD with two other public receipt sets. SROIE labels the shop, the date and the total, a better match, but only copies of it state a licence, not its creators. WildReceipt states none. CORD is the only one whose creators say how it may be used.

What the labels say. CORD labels every printed word with its box, and the receipt with its own classes. Only some of them mean the same as a field of the course’s contract:

Contract field	CORD label	Labelled on the 60 test receipts
total	total.total_price	58
subtotal	sub_total.subtotal_price	41
tax	sub_total.tax_price (two tax lines are added up)	27
currency	none: every receipt is Indonesian, so IDR	60
items	menu: name, count and line price	146 lines
vendor, date, order number, payment method	removed by CORD	not scored

CORD removed the shop and payment labels and blurred shop names in the pictures, for legal reasons in Indonesia. So four fields of the contract cannot be scored at all. That is the honest scope: 186 labelled values on the test receipts.

The answer sheet has traps. Read the labels before you trust them:

• Some printed totals have no label. cord-test-037 prints TOTAL 20.000, and CORD has no total for it. A field without a label is not scored.
• Two labels sit on another line: cord-test-018 and cord-test-071 label a “Sub Total” line as the total.
• Real receipts break the course’s adds-up rule. Check CORD’s own labels with it: total = subtotal + tax. Of the 41 receipts that the rule can check, only 24 add up. The others have a service charge, a discount, a tax that is already in the prices, or a total rounded to 500 rupiah. The labels are right; the rule is Canadian.

Set up the project

1. Download the starter project from the box at the top of this page, and unzip it into your projects folder. You get the folder cord-receipts. Open it in VS Code (File > Open Folder), then open a terminal (Terminal > New Terminal).

2. Make a virtual environment, install the packages, get the sample and run the tests:

   macOS / Linux

   python3 -m venv .venv
   source .venv/bin/activate
   python -m pip install -r requirements.txt
   python -m cord_eval.data
   python -m pytest -q

   Windows (PowerShell)

   py -m venv .venv
   .venv\Scripts\Activate.ps1
   python -m pip install -r requirements.txt
   python -m cord_eval.data
   python -m pytest -q

   Output (end)

   Downloading https://github.com/Nextia-Intelligence/nextia-learning-labs/raw/main/C13/read-real-receipts/data/cord-v2-test-sample.zip
   Checked cord-v2-test-sample.zip: SHA-256 2dd48e2cc141... is correct
   data/cord-v2-test-sample: 100 receipts, every file matches SHA256SUMS
   15 passed in 1.36s

3. Look at the sample, and at one receipt as the contract sees it:

   python -m cord_eval.sample
   python -m cord_eval.sample cord-test-007

   Output

   100 receipts in data/cord-v2-test-sample: dev 40, test 60
     total      labelled on  95  (dev 37, test 58)
     subtotal   labelled on  65  (dev 24, test 41)
     tax        labelled on  41  (dev 14, test 27)
     service    labelled on  12  (dev 4, test 8)
     discount   labelled on   6  (dev 3, test 3)
     item lines   246  (dev 100, test 146)
   cord-test-007  images/cord-test-007.jpg  (CORD test image 7, our split: dev)
     picture 576 x 864 px (CORD's original 576 x 864)
     total      111000
     subtotal   100909
     tax        10091
     service    (not labelled)
     discount   (not labelled)
     currency   IDR
       item: ? x Kupon 15 = 100000

What this tells us. The download’s checksum is right, and so is every file in it. The sample has 40 dev receipts and 60 test receipts, chosen at random with a fixed seed. Dev receipts are for looking at and changing the rules. Test receipts stay sealed until the rules are frozen. It is the train and test idea, for rules instead of a model: you cannot measure on what you tuned on.

Frame the problem

Arif writes the measure down before any run, so that no result can move it:

• The unit is one receipt photo. The values are the labelled total, subtotal, tax and currency: 186 on the test receipts.
• Each value is correct, missing or wrong, as in Field-level metrics. Rupiah are compared as numbers, so 91000 and 91000.00 agree.
• A receipt is complete when its labelled total, subtotal and tax are all correct.
• The safety measure is the number of wrong totals that go straight through, with no review. Dewi’s team pays from the total.
• The cost is the model’s calls (recorded, at list price) and the time and memory of OCR.

Part 1: the pipeline as it is

Run the course’s pipeline, unchanged, on the 100 photos. as_is reads each photo the way the course does: prepare it, run OCR, apply doc_extract/fields.py. OCR takes about a minute, and its results go to a cache.

python -m cord_eval.run --system as_is
python -m cord_eval.score --split dev --systems as_is

Output (end of the run, then the score)

[ 99/100] cord-test-098 (dev) total -
[100/100] cord-test-099 (dev) total -
as_is: 100 receipts -> out/cord/as_is.jsonl
dev receipts: 40. Correct values of the values CORD labels (rupiah compared as numbers).
system              total     subtotal          tax     currency         all  complete     items
as_is                2/37         1/24         0/14         0/40       3/115      2/40     0/100

wrong / missing per field (w/m):
  as_is       total 2/33  subtotal 0/23  tax 0/14  currency 0/40   total's box on CORD's label: 2 of 2

What this tells us. The pipeline read 3 of 115 labelled values on the dev receipts, and two of the totals it gave were wrong. Look at one receipt to see why. OCR read its rows well; the rules found nothing in them.

The same OCR lines, two sets of rules. The course's rules find nothing on cord-test-008; the rupiah rules find every value, and their boxes cover the words that CORD labelled.

Text description of the diagram

The photo shows the bottom of a receipt: three items, then Sub Total 45,455, PB1 (10%) 4,545, Rounding 0 and Total 50,000. The rupiah rules found the subtotal, the tax and the total, and their green boxes cover the rows where CORD labelled the amounts. The course’s rules found none of them, and no currency. The rupiah rules inferred IDR.

Three things stop the course’s rules:

• Money. contract.money() looks for two decimals, as Canadian receipts print them: 1011.78. Rupiah have no cents, and the thousands are grouped with a point or a comma: 50,000 or 60.000.
• Words. The rules know Subtotal, GST, HST and TOTAL. Indonesian receipts print Sub Total, PB1 or Pajak (restaurant tax), PPN, Service and GRAND TOTAL.
• Currency. The contract allows CAD and USD only.

None of this is OCR’s fault. The pipeline is a set of decisions about one country, written into its contract.

Part 2: read rupiah

Recipes in cups, recipes in grams

A cook from Toronto opens a recipe from Jakarta: “250 g flour”. Her measuring cup is useless, but her cooking is not. She changes one tool, the scale, and cooks as before. If she read “250” as cups, the cake would fail, and she might blame the oven.

The idea: a contract has a locale

The course’s contract says what each value means: money with two decimals, CAD or USD, tax lines named GST or HST. Those choices are a locale: the conventions of one country for numbers, dates, currency and words. When the locale changes, the contract changes first, and then the rules follow. Here the new money reader is cord_eval/idr.py. A group of three digits is thousands; a last group of exactly two digits is cents.

Printed	contract.money()	idr.amount()
100,909	none	100909
60.000	none	60000
Rp.118.000	none	118000
91000	91000.00	91000
9.500,00	9500.00	9500
TAX 10.00 % 4.664	10.00	4664

The course’s reader works when a receipt prints cents, as 9.500,00 does. Without cents, it is silent. Look at the last row: it reads the tax rate as the tax. A reader that is wrong is worse than one that is silent. The rupiah reader skips percentages and the Rp sign.

money_only uses the rupiah reader with the course’s words. It also pairs each label with the amount on the same printed row, along the photo’s slope, as the course’s rows() does. On a crooked photo, the amount at the right edge sits higher or lower than its label.

python -m cord_eval.run --system money_only --quiet
python -m cord_eval.score --split dev --systems as_is,money_only

Output

money_only: 100 receipts -> out/cord/money_only.jsonl
dev receipts: 40. Correct values of the values CORD labels (rupiah compared as numbers).
system              total     subtotal          tax     currency         all  complete     items
as_is                2/37         1/24         0/14         0/40       3/115      2/40     0/100
money_only          23/37         8/24         3/14        27/40      61/115     14/40    85/100

wrong / missing per field (w/m):
  as_is       total 2/33  subtotal 0/23  tax 0/14  currency 0/40   total's box on CORD's label: 2 of 2
  money_only  total 1/13  subtotal 0/16  tax 0/11  currency 0/13   total's box on CORD's label: 23 of 23

What this tells us. One change takes the pipeline from 3 to 61 of 115. Totals gain most, because many receipts print a plain TOTAL, a word that the course already knows. Subtotals and taxes stay low: their words are Indonesian.

Part 3: the Indonesian words, and the raw photo

Your turn: add the words

Open cord_eval/rules_id.py. Its LABELS list holds the course’s words only. Look at the OCR lines of the dev receipts, and add what they print:

python -m cord_eval.lines cord-test-007

Output

cord-test-007 (dev): 12 lines, 0.21 s
  1.00 [   67   523   158   553]  Kupon 15
  0.99 [  415   530   500   567]  100,000
  1.00 [   67   550   226   585]  ADD CHICKEN BOX
  1.00 [  459   563   500   591]  909
  0.94 [   61   606   153   640]  Subtetal
  1.00 [  417   620   497   651]  100,909
  0.93 [   60   636   162   669]  PB1(107)
  0.98 [  427   649   499   683]  10,091
  1.00 [   55   691   120   729]  Total
  1.00 [  413   703   495   738]  111,000
  0.99 [   53   754   109   788]  Cash
  1.00 [  382   758   494   794]  Rp.111,000

Every word must come from a dev receipt. If you add a word after you look at a test receipt, the test is no longer fair. The finished project’s list:

cord_eval/rules_id.py (part of the finished project)

LABELS = (
    ("subtotal", re.compile(r"^sub\W*t")),  # Sub Total, SUBTOTAL, SUB-TOTAL, SUBTTL, Sub, Total
    ("tax", re.compile(r"^(tax|vat|ppn|pb\W*1|pajak|restaurant tax)")),
    ("service", re.compile(r"^(service|svc|serv\b)")),
    ("discount", re.compile(r"^(discount|diskon|disc\b)")),
    ("total", re.compile(r"^(grand\W*total|total|tl\b)")),
    ("cash", re.compile(r"^(cash|tunai)")),
    ("change", re.compile(r"^(change|kembali)")),
)
NOT_A_TOTAL = re.compile(r"(item|qty|disc|tax|pajak)")  # "TOTAL ITEMS 3", "TOTAL DISC"

Two details come from dev receipts too. ^sub\W*t also matches OCR’s Subtetal (cord-test-007 above). And when a receipt prints both TOTAL and GRAND TOTAL, the grand total is the total and the plain TOTAL is the subtotal (cord-test-095).

Prepared or raw?

The course prepares every picture before OCR: crop to the paper, turn, straighten, enlarge. Image preparation measured it on the course’s documents. It fixed the turned scans and did not help the photos. Here every receipt is a photo, so measure both on the dev receipts:

python -m cord_eval.run --system adapted --quiet
python -m cord_eval.run --system adapted_raw --quiet
python -m cord_eval.score --split dev --systems as_is,money_only,adapted,adapted_raw

Output

adapted: 100 receipts -> out/cord/adapted.jsonl
adapted_raw: 100 receipts -> out/cord/adapted_raw.jsonl
dev receipts: 40. Correct values of the values CORD labels (rupiah compared as numbers).
system              total     subtotal          tax     currency         all  complete     items
as_is                2/37         1/24         0/14         0/40       3/115      2/40     0/100
money_only          23/37         8/24         3/14        27/40      61/115     14/40    85/100
adapted             31/37        21/24        13/14        35/40     100/115     30/40    81/100
adapted_raw         33/37        23/24        13/14        38/40     107/115     31/40    86/100

wrong / missing per field (w/m):
  as_is       total 2/33  subtotal 0/23  tax 0/14  currency 0/40   total's box on CORD's label: 2 of 2
  money_only  total 1/13  subtotal 0/16  tax 0/11  currency 0/13   total's box on CORD's label: 23 of 23
  adapted     total 0/6  subtotal 0/3  tax 0/1  currency 0/5   total's box on CORD's label: 31 of 31
  adapted_raw total 1/3  subtotal 0/1  tax 1/0  currency 0/2   total's box on CORD's label: 33 of 33

Dev receipts only (115 labelled values). Each bar adds one change: the money reader, then the words; the last two differ only in the picture that OCR reads.

Text description of the chart

On the 40 dev receipts, the pipeline as it is read 2.6% of the labelled values. With the rupiah reader it read 53.0%. With the Indonesian words, it read 87.0% on prepared pictures and 93.0% on the raw photos.

What this tells us. The words take the rules from 61 to 100 correct values. And the raw photo beats the prepared one: 107 against 100. Preparation’s crop looks for bright paper on a darker table. On real photos it sometimes keeps the wrong part (cord-test-022, cord-test-054). This decides the combined path before any test receipt is scored: it reads the raw photo.

Now freeze the rules. In a real project, commit them now, so that anyone can check that the test came after.

Part 4: the model where the rules stop

The rules still miss some totals. The course’s answer is from Combine methods: rules first, and the vision model only when needed. Here the combined path asks the model when the total is missing or the amounts do not add up. The adds-up check is the Indonesian one in cord_eval/check.py. It accepts a service charge, a discount, and a tax that is already in the prices.

The prompt changed too, because a prompt is part of the contract. Prompt v2-idr is the course’s prompt v2 with the same schema. It adds three rules: money in whole rupiah, currency IDR, and the Indonesian names of tax and total. Before Arif wrote it, he sent the course’s prompt v2, unchanged, with 5 receipts. The model read every labelled amount right. It wrote the currency as “absent, not printed”: the prompt allows only CAD or USD, and the model did not invent one. It did not guess, but the contract still had no place for rupiah.

The model’s answers were recorded: 21 calls with prompt v2-idr and 5 with v2, US$0.30 in all. The project replays them, so you need no key.

python -m cord_eval.vision_id cord-test-013

Output

recorded answer (replayed; nothing was sent)
cord-test-013, prompt v2-idr: gpt-6.1-sol-2026-09-29, 2293 tokens in, 731 out, 11.786 s, about US$0.0119
  total      51300
  subtotal   46636
  tax        4664
  currency   IDR

Part 5: the fair comparison

Every system ran on the same 60 test receipts, with the same labels and the same scorer, once:

python -m cord_eval.run --system all --quiet
python -m cord_eval.score --split test

Output

as_is: 100 receipts -> out/cord/as_is.jsonl
money_only: 100 receipts -> out/cord/money_only.jsonl
adapted_raw: 100 receipts -> out/cord/adapted_raw.jsonl
adapted: 100 receipts -> out/cord/adapted.jsonl
vision: 100 receipts -> out/cord/vision.jsonl; model answers used 21 (US$0.240 when recorded)
combined: 100 receipts -> out/cord/combined.jsonl; model answers used 21 (US$0.240 when recorded)
test receipts: 60. Correct values of the values CORD labels (rupiah compared as numbers).
system              total     subtotal          tax     currency         all  complete     items
as_is                3/58         1/41         0/27         0/60       4/186      2/60     3/146
money_only          31/58        22/41         6/27        43/60     102/186     15/60   110/146
adapted_raw         41/58        31/41        18/27        56/60     146/186     33/60   111/146
adapted             34/58        30/41        15/27        48/60     127/186     26/60   108/146
combined            53/58        34/41        22/27        60/60     169/186     47/60   111/146

wrong / missing per field (w/m):
  as_is       total 4/51  subtotal 1/39  tax 0/27  currency 0/60   total's box on CORD's label: 3 of 3
  money_only  total 4/23  subtotal 1/18  tax 0/21  currency 0/17   total's box on CORD's label: 31 of 31
  adapted_raw total 4/13  subtotal 0/10  tax 1/8  currency 0/4   total's box on CORD's label: 41 of 41
  adapted     total 4/20  subtotal 1/10  tax 1/11  currency 0/12   total's box on CORD's label: 34 of 34
  combined    total 4/1  subtotal 0/7  tax 1/4  currency 0/0   total's box on CORD's label: 44 of 53
  combined: model called for 21 receipts; to review 8, straight through 52; wrong totals caught by review 1, sent on 3

The test receipts, scored once (186 labelled values). The combined path is the rupiah rules on the raw photo, with the recorded model on 21 receipts.

Text description of the chart

On the 60 test receipts, the pipeline as it is read 2.2% of the labelled values. The rupiah reader alone read 54.8%. The rupiah rules read 68.3% on prepared pictures and 78.5% on the raw photos. The combined path read 90.9%.

What this tells us.

• The dev numbers were too kind. The rules on the raw photo fell from 107 of 115 (93%) on the dev receipts to 146 of 186 (78.5%) on the test receipts. The test receipts print words that no dev receipt showed (NET SALES, AMOUNT DUE, DUE), and some photos are harder. Without the sealed test, Dewi would have heard “93%”.
• Raw beat prepared again, 146 against 127, so the dev decision holds.
• The combined path read 169 of 186 (90.9%) and completed 47 of 60 receipts. The model was called for 21 receipts: US$0.24 in all, about US$0.004 per receipt over the 60.
• The boxes are right. Every total that the rules read has its box on the word that CORD labelled: 41 of 41. A model value gets a box only when grounding finds it on an OCR line. Of the 53 correct combined totals, 44 have their box on the labelled word (more below).

How good is the model alone? Compare the three systems on the 21 receipts where the rules gave up:

The 21 test receipts where the total was missing or did not add up (65 labelled values). The combined path keeps the rules' value when the two disagree, and sends the receipt to review.

Text description of the chart

On the 21 receipts sent to the model, the rules alone read 60.0% of the labelled values. The model alone read 98.5%, and the combined path 95.4%.

On these receipts the model read 64 of 65 values, and the combined path 62. The combined path is a little lower on purpose. When the rules and the model disagree, it keeps the rules’ value and sends the receipt to a person. A disagreement is a question, not a vote. To send every test receipt to the model would cost about US$0.67 (US$0.0112 per call, median). It would also give 60 answers with no box of their own, at about 11 seconds each.

Now open notebooks/evaluate-cord.ipynb in VS Code, with the project’s .venv as its kernel. It reads your runs, makes the same tables and charts, and ends with three checks that hide their answers.

What fails, and why

Of the 60 test receipts, the combined path sent 8 to review and 52 straight through. Review caught one wrong total. Three wrong totals went straight through. Dewi needs to hear that number first.

Four failures from the runs. Preparation cropped or turned real photos wrongly (1, 2); OCR misread a faded total (3); grounding put a right value on the first line that shows it (4).

Text description of the diagram

Four failures from the real runs. First, preparation’s crop kept a small corner of a dark photo, so nothing was left to read. Second, preparation cropped an upright receipt and then turned it 90 degrees. Third, OCR read a faded total line as “Total 48:888” instead of 40,000. Fourth, the model’s total, 13.000, was grounded on the first line that shows 13.000, an item line, not the Total line.

Common mistake

A wrong total that nothing can check

cord-test-089 is a faded receipt. OCR read its total line as Total 48:888, and the rules took 888. The receipt prints no subtotal, so the adds-up check had nothing to compare. The total looked fine.

python -m cord_eval.show cord-test-089

cord-test-089 (test), system combined
  total     888       found      from rules  box [52, 749, 798, 861]    label 40000     wrong
  subtotal  -         absent     from rules  no box                     label 40000     missing
  tax       -         absent     from rules  no box                     label (no label) unlabelled
  currency  IDR       inferred   from rules  no box                     label IDR       correct
  -> straight through

The other two are alike. On cord-test-020 the rules read 377 for 377,859. On cord-test-050 they read 500 for 19,500. Neither receipt had a subtotal that the rules read.

Fix: send a receipt to review, or to the model, when its total cannot be checked. Or check the total against the item lines. Measure the fix on new receipts. Once you have studied their failures, these 60 are no longer a fair test.

The other failures, by cause:

• Preparation on real photos. Preparation cropped 82 of the 100 photos, straightened 32, enlarged 6 and turned 1. Its crop and turn were built for scans and generated photos; real tables, shadows and walls fool them.
• New words. NET SALES, AMOUNT DUE, DUE: no dev receipt printed them, so the model had to read those totals.
• Grounding on the first matching line, a limit named in Combine methods. A small receipt prints its total on the item, the subtotal and the total line. The value is right; its box is on the first of them.
• Unverified values. On cord-test-013 no OCR line showed the model’s three right amounts, so all three went to review: the safe answer.
• Rounded totals. On cord-test-059 the subtotal and tax add up to 680,339, and the total is 680,500. A person must accept that.

The recommendation

Arif’s answer to Dewi, in her words:

1. Use the combined path, not the pipeline as it is. As it is, it read 4 of 186 labelled values. The adapted rules read 78.5%, and with the model on about a third of the receipts, 90.9%.
2. Keep a person on every receipt that the pipeline cannot check. Today 8 of 60 go to review. Also send the receipts whose total cannot be checked against a subtotal or the items. That closes the gap that let three wrong totals through, at the cost of more reviews: measure the new rate on next month’s receipts.
3. Turn preparation off for phone photos, or fix the crop first, then measure again.
4. Budget the model at about one US cent per call, for about a third of the receipts. That is about US$0.004 per receipt at list price. Prices change: check the current price page before you plan.
5. Do not use it for the shop’s name, the date or the payment method yet. Nothing here measured them.

End to end: one receipt

Follow one test receipt through the whole path. cord-test-059 is a restaurant bill with a large printed total:

python -m cord_eval.show cord-test-059

Output

cord-test-059 (test), system combined
  model asked: the amounts do not add up
  total     21        found      from rules  box [802, 1542, 1150, 1616] label 680500    wrong
            conflict: the rules read 21, the model read 680500
  subtotal  618490    found      from both   box [183, 1662, 1158, 1745] label 618490    correct
  tax       61849     found      from model  box [991, 1741, 1156, 1806] label 61849     correct
  currency  IDR       inferred   from rules  no box                     label IDR       correct
  -> review: the amounts do not add up; total: conflict: the rules read 21, the model read 680500

What this tells us. OCR misread the total line, and the rules took 21. The amounts did not add up, so the combined path asked the model. The model read 680500, the labelled total. The rules and the model disagreed, so the receipt went to review with both values, and a person decides. The subtotal came from both readers, and the tax from the model, grounded on its line. Nothing wrong reached the payment.

A receipt that the rules read whole, like cord-test-014, goes straight through with a box on each value.

Time and memory

On the author’s computer, OCR of a raw photo took 0.27 seconds per receipt (median). A prepared one took 0.56 seconds, because preparation reads a small copy first. The model took about 11 seconds per call.

Memory surprised Arif. One process that read the 100 prepared photos peaked at 2,515 MB; on the raw photos, at 1,250 MB. Both runs used the course’s OCR_MAX_SIDE of 1,280 pixels. On the course’s own pages, the peak was 1,447 MB, under the course’s limit of 2 GB. So a worker that prepares real photos needs more memory than the course planned: one more reason to read the raw photo.

Insights and limits

• A pipeline is a set of decisions about one place. OCR read the Indonesian receipts well. The contract’s money format, words and currency list failed. Change the contract first, then the rules.
• Measure on receipts that you did not tune on. The dev score said 93%; the sealed test said 78.5%.
• Validation needs something to compare. Three wrong totals had no subtotal next to them.
• Limits. 60 test receipts, so one receipt moves a score by about 1.7 points. One country, mostly restaurants, four fields not scored. The sample’s smaller pictures cost 4 of 301 labelled values against CORD’s originals (253 against 257). Windows and Linux are not tested.

Responsible use

• Credit the data, and keep its changes listed. Write “CORD v2 by NAVER CLOVA, CC BY 4.0”, and keep the dataset card next to your results.
• Real receipts are real people’s purchases. The creators blurred shop names and card numbers. Do not try to identify a shop or a person, and do not join these receipts with other data.
• Expense claims are about money and trust. A wrong total can pay someone too much or too little. Keep a person on every receipt that the pipeline cannot check, and say so in the claim process.
• Do not promise one country’s numbers in another. A new country, shop chain or phone camera needs a new measurement.

Make it a portfolio piece

This project shows an employer that you can take a working pipeline into a new setting and measure it honestly. It has labels, a sealed test, one change at a time, and failures with their causes. Make it yours with one change, then publish it.

What to put in the repository:

cord-receipts/
  README.md                       the problem, the data and licence, the approach, the results, the limits
  DATASET.md                      the dataset card: source, CC BY 4.0, changes, checksums
  LICENSE                         MIT, for the code
  cord_eval/                      the label mapping, the rupiah rules, the runs, the scorer
  doc_extract/                    the course's pipeline, unchanged
  recordings/                     the model's recorded answers (no key in them)
  notebooks/evaluate-cord.ipynb   run from top to bottom, with its outputs
  docs/results.md                 your numbers: dev, then test, with the review routing
  tests/                          the tests

Do not commit the sample itself: python -m cord_eval.data gets it with its checksum.

How to describe it in an interview, in three sentences:

“I measured an OCR pipeline built for Canadian receipts on 100 real Indonesian receipts, against their published labels. I adapted it on 40 receipts. One test on 60 then gave 79% of labelled values, and 91% with a vision model on a third of them. Three wrong totals still went straight through, and I can show why.”

Go further:

• Add the review rule for totals that cannot be checked, and measure it on CORD’s validation split (100 new receipts).
• Check the total against the sum of the item lines, and count the wrong totals that it catches.
• Fix preparation’s crop for photos, and measure raw against prepared again.
• Ground the model’s total on a line with a total word, not on the first matching line, and count the boxes on the label.

Knowledge check

Check your understanding

The rupiah rules on the raw photo scored 93% on the dev receipts and 78.5% on the test receipts. Dewi asks which number to believe, and why the two differ.

Show the answer

Believe the test number. The rules were written while looking at the dev receipts: every word and every special case came from them. So the dev score measures how well the rules fit those 40 receipts. The test receipts were scored once, after the rules were frozen. They show words that no dev receipt had (NET SALES, AMOUNT DUE, DUE) and harder photos, as new receipts will. With 60 receipts the test number is still rough: one receipt moves it by about 1.7 points.

Check your understanding

On the 21 receipts sent to the model, the model alone read 64 of 65 values and the combined path 62. Sam says: 'Then drop the rules and send every receipt to the model.' What do you answer?

Show the answer

Those 21 are the receipts where the rules failed, so the comparison flatters the model. The combined path scores lower there on purpose. When the rules and the model disagree, it keeps the rules’ value and sends the receipt to review, so a person decides. To send all 60 receipts to the model costs about US$0.67 instead of US$0.24. It takes about 11 seconds per receipt instead of a quarter of a second, and it gives values with no box of their own. The rules read most receipts right, with the box on the labelled word, at no cost.

Check your understanding

Three wrong totals went straight through the combined path, although it has an adds-up check. Explain why, and propose a change that you can measure fairly.

Show the answer

On each of them, OCR misread or cut the total: 888 for 40,000, 377 for 377,859, 500 for 19,500. And the rules read no subtotal. The adds-up check needs a subtotal and a total, so it could not check anything. Nothing else flagged the receipt. A change: send every receipt whose total cannot be checked to review or to the model. Or compare the total with the sum of the item lines. Measure it on new receipts, such as CORD’s validation split. You used these 60 test receipts to find the problem.

Recap

• You mapped CORD’s labels onto the course’s contract, honestly: total, subtotal, tax and currency, 186 values on the test receipts. Vendor, date and payment could not be scored.
• The pipeline as it is read 4 of 186. Its contract assumed cents, Canadian words and CAD or USD.
• On the 40 dev receipts, one change at a time: the rupiah reader, the Indonesian words, the raw photo instead of the prepared one. Then the rules were frozen.
• On the 60 test receipts, scored once: 78.5% with the rules. With the recorded model on 21 receipts: 90.9%, at US$0.24 in all.
• Eight receipts went to review. Three wrong totals went straight through, because no subtotal was there to check them. That is the next fix, measured on new receipts.

Next, the second case study: recover a table that runs over two pages of a public report, and check it cell by cell.

Next: Extract a table from a public report