Field-level metrics

What you will be able to do

Measure a document reader the way its users feel it: field by field, by document type, with the kind of each error. Count exact and normalised matches apart, and count the documents that are complete. One accuracy number hides the errors that matter most, such as invented values.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• The extraction contract and its allowed reasons for an absent value, from Define the extraction contract.
• The six ways to read a receipt that this course compares. They are the text layer, OCR raw and prepared, the vision model with two prompts, and the combined pipeline. They come from Text and OCR paths and Combine methods.
• Accuracy and slices, from Prediction metrics and Uncertainty and slices in Evaluating and Testing AI Systems, if you took it. This lesson explains every count it uses.

What you need on your computer

• Your project at the end of the previous module: python -m pytest -q prints 98 passed. If your project is broken, copy the snapshot end-of-m05, as the snapshots README says.
• The labs repository in ~/projects/nextia-learning-labs, up to date.

Tested with Python 3.12.13 and RapidOCR 3.10.0 on macOS (Apple silicon). The Windows and Linux commands are not tested. Every document in this course is made by the course’s generator; no real customer or shop is in it.

Get the lesson’s file

1. In the project folder, copy the module and its test from the snapshot:

   macOS / Linux

   S=../nextia-learning-labs/C13/snapshots/end-of-m06
   cp $S/doc_extract/evaluate.py doc_extract/
   cp $S/tests/test_evaluate.py tests/

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C13\snapshots\end-of-m06"
   Copy-Item "$S\doc_extract\evaluate.py" doc_extract\
   Copy-Item "$S\tests\test_evaluate.py" tests\

   Copy only these two files. The snapshot also has the files of the next two lessons, and the last lesson changes one file of the rules.

2. Run the tests:

   python -m pytest -q

   The last line is 101 passed. The tests read a few photos with OCR, so the first run can take about 30 seconds.

Mark a form box by box

A teacher marks a job application form. The form has nine boxes: name, date of birth, address, and so on. The applicant can fail a box in three ways. They can leave it empty. They can write a wrong answer. Or they can write something in a box that must stay empty, such as a middle name that they do not have. At the end, the teacher asks one more question: “Can the office use this form as it is?” One bad box in a required field, and the answer is no.

A score such as “92% of the boxes are right” hides all of this. It does not say which boxes failed, or how. It does not say how many forms the office can use. A score for a document reader has the same problem.

The problem: one number picks the wrong reader

The course measured six ways to read the 54 receipt documents. The receipts print 435 values of the nine contract fields. Here is the simplest score, the share of those printed values that each reader got right:

Reader	Printed values right
Text layer + rules	169 of 435 (38.9%)
OCR on the raw picture + rules	392 (90.1%)
OCR on the prepared picture + rules	408 (93.8%)
Vision model, plain prompt (v1)	424 (97.5%)
Vision model, precise prompt (v2)	425 (97.7%)
Combined pipeline	427 (98.2%)

Tomás reads the table and makes a choice: “The model with the plain prompt gets 97.5%. That beats OCR by almost four points. We can use the shorter prompt.” Priya, who owns the evaluation, is not sure. She remembers Input and prompt design: the plain prompt filled empty order numbers with the till’s transaction number.

The score above cannot show that problem. It counts only values that the receipt prints. An order number that the receipt does not print is not in the 435, so an invented one is never counted. The score also mixes two different failures. A reader that says “I cannot read this” loses points, and so does a reader that reads a wrong value. The text layer gets 38.9% because it cannot read pictures at all, but it never gives a wrong value.

The idea: one outcome per field, then count by slice

Field-level evaluation compares every field of every document with its true value, and gives it one of five outcomes. The first question is the teacher’s empty box: “Does the document print this field?”

The five outcomes of one field. Each example is a real case from the course's documents.

Text description of the diagram

A decision tree for one field of one receipt. The first question is whether the document prints the field. If it does, a system that gives no value has a missing value, as on the unreadable photo D046. A value that is different after normalising is wrong, as D009’s LARKEFIELD. A value that is the same is correct: exact when it is also written the same way (Larkfield), normalised when it is not (LARKFIELD). If the document does not print the field, a system that gives no value is correct absent, as the cut-off total of D045. A system that gives a value has invented it, as 0693-220 on D003.

• correct: the document prints the value, and the system gives the same value after normalising.
• missing: the document prints the value, and the system gives none.
• wrong: the system gives another value.
• correct absent: the document does not print the field, and the system says so.
• invented: the document does not print the field, and the system gives a value anyway.

Normalising means writing a value in one standard form before you compare it. The contract’s normalise turns 1 011,78 into 1011.78 and LARKFIELD into Larkfield. An exact match also needs the same writing. Both counts are useful, for different users.

Then you count the outcomes in slices: by document type and by field. A slice is a part of the data that you count on its own. A table of these counts is an evaluation matrix. Last, you ask the office’s question for each document. A document is complete when every required field is correct and nothing is invented.

A tiny example: five fields by hand

Start python in the project folder and type:

from doc_extract import evaluate
cases = [("total", "211.68", "211.68"), ("total", None, "211.68"), ("total", "211.86", "211.68"),
         ("order_number", "0693-220", None), ("order_number", None, None)]
for name, got, want in cases:
    print(f"{name:<13} read {str(got):<9} true {str(want):<7} -> {evaluate.outcome(name, got, want)}")

total         read 211.68    true 211.68  -> correct
total         read None      true 211.68  -> missing
total         read 211.86    true 211.68  -> wrong
order_number  read 0693-220  true None    -> invented
order_number  read None      true None    -> correct_absent

None means “no value”. In the truth, None means that the receipt does not print the field. Read the last two lines together. The same empty truth gives “invented” or “correct absent”, and only the system’s answer decides which.

Now score two real documents with five systems. D003 is an in-store receipt with no order number. D045 is a photo whose bottom is cut off:

from doc_extract import documents, evaluate
systems = ("text", "ocr_prepared", "vision_v1", "vision", "combined")
for doc in ("D003", "D045"):
    r = evaluate.evaluate(systems, [documents.get(doc)])
    rows = {s: r["systems"][s]["per_doc"][doc] for s in systems}
    print(doc + " " * 12 + "".join(f"{s:>15}" for s in systems))
    for name in rows["combined"]["fields"]:
        print(f"  {name:<14}" + "".join(f"{rows[s]['fields'][name]:>15}" for s in systems))
    print(f"  {'complete':<14}" + "".join(f"{str(rows[s]['complete']):>15}" for s in systems))

D003                       text   ocr_prepared      vision_v1         vision       combined
  vendor                correct        correct        correct        correct        correct
  order_number   correct_absent correct_absent       invented correct_absent correct_absent
  date                  correct        correct        correct        correct        correct
  currency              correct        correct        correct        correct        correct
  subtotal              correct        correct        correct        correct        correct
  shipping       correct_absent correct_absent correct_absent correct_absent correct_absent
  tax                   correct        correct        correct        correct        correct
  total                 correct        correct        correct        correct        correct
  payment_method        correct        correct        correct        correct        correct
  complete                 True           True          False           True           True
D045                       text   ocr_prepared      vision_v1         vision       combined
  vendor                missing        correct        correct        correct        correct
  order_number          missing        correct        correct        correct        correct
  date                  missing        correct        correct        correct        correct
  currency              missing        missing        correct        correct        correct
  subtotal              missing        correct        correct        correct        correct
  shipping       correct_absent correct_absent correct_absent correct_absent correct_absent
  tax                   missing        correct        correct        missing        correct
  total          correct_absent correct_absent correct_absent correct_absent correct_absent
  payment_method correct_absent correct_absent correct_absent correct_absent correct_absent
  complete                False          False           True          False           True

D003 and D045, scored field by field for five systems. On D003, only the plain prompt fails, and it fails by inventing. On D045, three systems fail in three different places.

Text description of the diagram

On the left of each card, the page with green boxes around the evidence of the combined pipeline. On D003, an orange box marks the till’s transaction number, 0693-220, which prompt v1 gave as the order number. On the right, a grid of outcomes: fields down, systems across. D003: all systems are right on every printed field and keep order number and shipping absent, except Model v1, which invents the order number. So Model v1 is the only system whose D003 is not complete. D045: Text cannot read a photo and misses six fields. OCR prepared misses the currency, and Model v2 misses the tax. The total and the payment line are below the cut edge for everybody. Model v1 and the combined pipeline are complete.

On D003, the plain prompt reads all seven printed values right. It would score 100% on the simple measure. But it invents the order number, so the document is not complete: an agent would search for an order that does not exist.

On D045, the total and the payment are not printed, and every system leaves them empty: “correct absent”. This is the right answer for a cut-off photo. The text layer misses six values, because a photo has no text layer: missing, not wrong. OCR and the precise prompt each miss one value, and the combined pipeline fills both gaps.

Exact or normalised: who uses the value?

The two counts measure the same answers in two ways. The course counted the values that are right after normalising but are not written as the truth. This is a check over the whole set, run by the course:

== vision_v1: 137 values right but not written exactly as the truth
   21 x vendor          read 'LARKFIELD Home & Garden'    true 'Larkfield'
   14 x shipping        read '0.0'                        true '0.00'
   10 x payment_method  read 'Mastercard ****1907'        true 'Mastercard'
    9 x payment_method  read 'Debit ****3310'             true 'Debit'
    7 x payment_method  read 'Amex ****2004'              true 'Amex'
    6 x vendor          read 'LARKFIELD'                  true 'Larkfield'
== vision: 32 values right but not written exactly as the truth
   13 x vendor          read 'LARKFIELD Home & Garden'    true 'Larkfield'
   13 x vendor          read 'LARKFIELD'                  true 'Larkfield'
    6 x vendor          read 'MAPLE & PINE'               true 'Maple & Pine Hardware'
== ocr_prepared: 6 values right but not written exactly as the truth
    6 x vendor          read 'MAPLE & PINE'               true 'Maple & Pine Hardware'

The model copies what the paper prints: LARKFIELD Home & Garden. The contract’s vendor list knows that this is Larkfield, so the value is right after normalising. Only 19 of the precise prompt’s 53 vendors are exact. The plain prompt is worse: it answers 0.0 for 0.00, and it keeps the card digits in Mastercard ****1907. The rules write the contract’s form at once, so 46 of their vendors are exact.

Which count matters depends on who uses the value. A refund needs the amount, and 189.0 is the same money as 189.00, so normalised is enough. A letter to the customer, a search by order number or a join with another system needs the exact text. So report both. And normalise in one place, the contract, so that the evaluation and the pipeline use the same rule.

The whole matrix

1. Evaluate the combined pipeline:

   python -m doc_extract.evaluate

   == combined: 427 of 435 printed values correct (98.2%); missing 8, wrong 0, invented 0
   field           digital_pdf    scan_pdf  scan_image       photo   mixed_pdf   exact
   vendor                21/21         7/7       10/10       13/14         1/1  46/53
   order_number          16/16         6/6         7/7       11/12         1/1  41/42
   date                  21/21         7/7       10/10       13/14         1/1  52/53
   currency              21/21         7/7       10/10       14/15         1/1  53/54
   subtotal              21/21         7/7       10/10       14/15         1/1  53/54
   shipping              10/10         2/2         3/3         4/4           -  19/19
   tax                   21/21         7/7       10/10       14/15         1/1  53/54
   total                 21/21         7/7       10/10       13/14         1/1  52/53
   payment_method        21/21         7/7       10/10       13/14         1/1  52/53
   complete documents: digital_pdf 21/21, all 53/54, scan_pdf 7/7, scan_image 10/10, photo 14/15, mixed_pdf 1/1
   item lines found: 99 of 105  (i = invented values)

   The first run reads every scan and photo with OCR, so it takes about 30 seconds. Each cell is “right / printed” for one field and one document type. The exact column counts all types together. The column mixed_pdf is the warranty claim, whose page 2 is a scanned receipt.

2. Evaluate all six systems:

   python -m doc_extract.evaluate --systems text,ocr,ocr_prepared,vision,vision_v1,combined

   The output has one block for each system. Here is the block of the plain prompt:

   == vision_v1: 424 of 435 printed values correct (97.5%); missing 8, wrong 3, invented 11
   field           digital_pdf    scan_pdf  scan_image       photo   mixed_pdf   exact
   vendor                21/21         7/7       10/10       13/14         1/1  21/53
   order_number       16/16+5i         5/6      7/7+3i    11/12+3i         1/1  40/42
   date                  21/21         6/7       10/10       12/14         1/1  50/53
   currency              21/21         7/7       10/10       14/15         1/1  53/54
   subtotal              21/21         7/7       10/10       14/15         1/1  14/54
   shipping              10/10         2/2         3/3         4/4           -   5/19
   tax                   21/21         7/7       10/10       14/15         1/1  44/54
   total                 21/21         7/7       10/10       13/14         1/1  45/53
   payment_method        21/21         7/7       10/10       13/14         1/1  15/53
   complete documents: digital_pdf 16/21, all 40/54, scan_pdf 6/7, scan_image 7/10, photo 10/15, mixed_pdf 1/1
   item lines found: 104 of 105  (i = invented values)

   +5i means 5 invented values in that cell. The 11 invented values are all order numbers. Only 14 of 54 subtotals are exact, because the plain prompt answers numbers such as 189.0 for 189.00.

The explorer shows the same matrix as a heat map, for each system. Before you click, predict.

Explore

Where does each reader fail?

Model, prompt v1: 424 of 435 printed values right (97.5%). Missing 8, wrong 3, invented 11. Complete documents: 40 of 54.

The plain prompt gets many values right, but it invents 11 order numbers. So only 40 documents are complete.

Model, prompt v1: printed values right, by field and document type

Field	Digital PDF 21 doc.	Scanned PDF 7 doc.	Scanned picture 10 doc.	Phone photo 15 doc.	Claim (mixed) 1 doc.	All types	Written exactly
vendor	21/21	7/7	10/10	13/141 missing	1/1	52/531 missing	21/53
order_number	16/16+5 invented	5/61 wrong	7/7+3 invented	11/121 missing +3 invented	1/1	40/421 missing 1 wrong +11 invented	40/42
date	21/21	6/71 wrong	10/10	12/141 missing 1 wrong	1/1	50/531 missing 2 wrong	50/53
currency	21/21	7/7	10/10	14/151 missing	1/1	53/541 missing	53/54
subtotal	21/21	7/7	10/10	14/151 missing	1/1	53/541 missing	14/54
shipping	10/10	2/2	3/3	4/4	not printed	19/19	5/19
tax	21/21	7/7	10/10	14/151 missing	1/1	53/541 missing	44/54
total	21/21	7/7	10/10	13/141 missing	1/1	52/531 missing	45/53
payment_method	21/21	7/7	10/10	13/141 missing	1/1	52/531 missing	15/53
Complete documents	16/21	6/7	7/10	10/15	1/1	40/54

Darker green: more printed values right. Orange words: wrong or invented values.

The 54 receipt documents of the course, scored field by field. Each cell: printed values right, then the errors by category. Measured with the project's evaluate command; the vision answers are the course's recordings.

Text description of the explorer

One control picks the system. The table shows the nine fields down and five document types across, then all types and the values written exactly. Each cell gives the printed values right, with a darker green for a higher share, and the errors: missing, wrong (orange) and invented (orange). The bottom row gives the complete documents. Text layer: 169 of 435 right, 266 missing, nothing wrong; it reads only digital PDFs; 19 of 54 complete. OCR raw: 392 right, 42 missing, 1 wrong; 45 complete. OCR prepared: 408 right, 27 missing; every scanned PDF complete; 46 complete. Model v1: 424 right, 8 missing, 3 wrong and 11 invented order numbers (5 on digital PDFs, 3 on scanned pictures, 3 on photos); 40 complete. Model v2: 425 right, 9 missing, 1 wrong (D009’s vendor), nothing invented; 19 of 53 vendors exact; 51 complete. Combined: 427 right, 8 missing (all on D046), nothing wrong or invented; 53 complete. The answer: OCR on prepared pictures gives 46 complete documents and the plain prompt 40. The difference is in the order number row, where the plain prompt invents 11 values.

Read the matrix of each system as a map of where to work next. The text layer is never wrong: its misses are the pictures that it cannot read. Preparation fixes the scans but not the photos. The precise prompt removes every invented value. The combined pipeline is complete everywhere except one photo.

Complete documents: the office’s question

The document count asks if an agent can use a receipt without checking it. Write the full result to a file:

python -m doc_extract.evaluate --systems vision_v1,vision,combined --json out/evaluation.json

The OCR results come from the cache now, so this takes about a second. Then list the documents that are not complete. In python:

import json
result = json.load(open("out/evaluation.json", encoding="utf-8"))
for system in ("vision_v1", "vision", "combined"):
    per_doc = result["systems"][system]["per_doc"]
    bad = [d for d, row in per_doc.items() if not row["complete"]]
    print(f"{system}: {len(per_doc) - len(bad)} of {len(per_doc)} complete; not complete: {', '.join(bad)}")

vision_v1: 40 of 54 complete; not complete: D003, D007, D013, D017, D018, D021, D028, D033, D034, D036, D042, D046, D048, D049
vision: 51 of 54 complete; not complete: D009, D045, D046
combined: 53 of 54 complete; not complete: D046

So the plain prompt is only 0.2 points behind the precise prompt on values, but 11 documents behind on completion. The values and the documents tell different stories, and the agents work with documents. In the combined pipeline, only D046 is not complete. Nobody can read it, and it goes to review with six missing fields, as Confidence and review routing showed.

Tables are measured cell by cell

Receipts have fields. A price list has a table, and a table has its own metric: the share of cells right, each one in its row and column. Tables and spreadsheets measured it. Run the two commands again and read the last line of each:

python -m doc_extract.tables D056
python -m doc_extract.tables D056 --naive

cells correct: 246 of 246 (100%)
45 rows (header rows counted as data), cells correct: 172 of 246 (70%)

The page-by-page reading puts the repeated header rows in as products, and it loses the categories of the merged cells. Every one of its values is on the page, so a count of values would hide the damage. The cell count shows it. The receipts’ item lines are a small table too: the combined pipeline finds 99 of 105 lines, and the model alone 104. Keep these counts apart from the nine fields, because a missed item line does not stop a refund.

When to evaluate field by field, and what it costs

• Evaluate against the truth, never against the system’s own output. Here the generator knows the truth by construction. The course author checked it by reading. For real documents, a person must label each field by hand, and that work is the main cost of an evaluation.
• Count every field, also the fields that are not printed. Otherwise invented values disappear from the score.
• Read small slices with care. The claim is one document, so 1/1 says almost nothing. A slice of 7 scanned PDFs moves by 14 points when one value changes.
• Do not average the fields into one number to choose a reader. Use the matrix to find the cells that fail, and the complete documents to compare systems.
• Keep the evaluation set fixed between versions. Otherwise you cannot compare the numbers.

Common mistake

The invented order numbers that scored as right

Symptom: A teammate writes a quick evaluation script. The plain prompt scores 97.5% and wins. Two weeks after the switch, agents report refunds that wait, because they search for order numbers such as 0693-220 that do not exist.

Cause: The script compared only the fields that the truth has. A field with no true value was skipped, so the 11 invented order numbers were never counted. This is a constructed story, but the numbers are the course’s real results for the plain prompt.

Diagnose: Count the outcomes per field, including invented. In the project, python -m doc_extract.evaluate --systems vision_v1 shows +5i, +3i and +3i in the order-number row. Then look at the complete documents: 40 of 54.

Fix: Score every field of every document with the five outcomes. Compare systems on complete documents and on the invented count, not only on printed values right.

Consequence in AI work: a metric decides what a team improves. A metric that cannot see a kind of error lets that error grow. This matters most for language models, which fill gaps with plausible text.

Your turn: does preparation help everywhere?

Change one condition: compare OCR on the raw picture with OCR on the prepared picture. Predict first. In which document type does preparation help most? Is there a field where it makes OCR worse? Then run:

python -m doc_extract.evaluate --systems ocr,ocr_prepared

Show the answer

Here are some cells from the two blocks, side by side:

                   ocr (raw)                         ocr_prepared
                   scan_pdf  scan_image  photo       scan_pdf  scan_image  photo
vendor               6/7       9/10      13/14         7/7      10/10      13/14
currency             6/7       9/10      14/15         7/7      10/10      13/15
subtotal             6/7       7/10      12/15         7/7       8/10      12/15
complete documents   6/7       7/10      12/15         7/7       8/10      11/15

Preparation helps the scans most. Two documents change completely: D025, a scanned PDF on its side, and D026, a scanned picture upside down. Raw OCR reads almost nothing on them, and preparation turns them upright, so every field becomes right. On photos, preparation does not help here. The currency drops from 14/15 to 13/15, because OCR on the prepared picture of D045 misses it. So one fewer photo is complete. Over all 54 documents, preparation still wins: 408 against 392 values right, and 46 against 45 complete documents. A slice shows what a total hides, in both directions.

See it end to endRead real receiptsCase study

The receipt case study scores six systems value by value on 60 real receipt photos that no rule was tuned on. It also counts the number that matters most to the business: the wrong totals that go straight through.

See it end to endExtract a table from a public reportCase study

The table case study scores five ways to read the same table cell by cell against the agency’s CSV: each cell is a match, wrong, missing or extra.

Knowledge check

Check your understanding

Explain why the plain prompt (v1) is worse than OCR on prepared pictures for Larkfield, although it gets more printed values right.

Show the answer

The printed-value score counts only fields that the receipt prints, so it cannot see invented values. The plain prompt invents 11 order numbers on receipts that print none. Each one makes its document not complete, and an agent would search for an order that does not exist. So the plain prompt gives 40 complete documents of 54, and OCR on prepared pictures gives 46. OCR misses some values, but a missing value goes to review with a reason. An invented value looks right.

Check your understanding

Predict: the model reads a receipt that prints Cash and 211.68, and answers payment_method 'CASH' and total 211.680. What are the outcomes, exact and normalised?

Show the answer

Both fields are correct after normalising: the contract writes CASH as Cash and 211.680 as 211.68. Neither is exact, because the text differs from the truth. So they count in the “correct” cells of the matrix, but not in the exact column. For a refund, normalised is enough. For a field that another system joins on, such as an order number, you need the exact form. So normalise before you store the value.

Check your understanding

Practical check: run python -m doc_extract.evaluate. Which document is not complete in the combined pipeline, which outcome do its fields get, and why is that acceptable?

Show the answer

D046, the unreadable photo. Its 8 printed values are missing: the pipeline gives no value, with the reason unreadable, and it invents nothing. This is acceptable because nobody can read the picture. The honest answer is “no value”, and the document goes to review, where a person asks the customer for a better picture. These 8 missing values are the combined pipeline’s only errors: 427 of 435 right, 0 wrong, 0 invented.

Recap

• Score every field of every document with one of five outcomes: correct, missing, wrong, correct absent, invented.
• Exact and normalised matches answer different needs. The model writes LARKFIELD: 19 of 53 vendors exact, 51 right after normalising.
• Count by document type and field. The evaluation matrix shows where each reader fails, and two slices can move in opposite directions.
• Complete documents answer the office’s question. The plain prompt is 97.5% right on values and 40 of 54 complete. The combined pipeline is 98.2% and 53 of 54.
• Tables are measured cell by cell: 246 of 246 for the price list, 172 with a page-by-page reading.

Next, you measure what each page costs in time, money, memory and storage, and plan a day of work.

Next: Operational constraints