Correction workflow

What you will be able to do

Record what a reviewer changes as a new row next to what the pipeline extracted, with a name, a note and a version. Nothing is erased, so anyone can see later what the pipeline read, who changed it, and why.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The evidence record of a value (file, page, box), from Provenance representation.
• The review queue and its reasons, from Confidence and review routing.
• A table, a row and a SQL SELECT, from SQL and Data Preparation for AI, if you took it. The lesson explains every statement it uses.

What you need on your computer

• Your project at the end of the previous lesson: python -m pytest -q prints 94 passed.
• Nothing to install: the database is SQLite, which comes with Python.

Tested with Python 3.12.13 on macOS (Apple silicon). The Windows and Linux commands are not tested. The reviewers’ corrections in this lesson are made-up review stories; the paper receipts’ text comes from the course’s generator.

Get the lesson’s file

1. In the project folder, copy the module and its test from the snapshot:

   macOS / Linux

   S=../nextia-learning-labs/C13/snapshots/end-of-m05
   cp $S/doc_extract/corrections.py doc_extract/
   cp $S/tests/test_corrections.py tests/

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C13\snapshots\end-of-m05"
   Copy-Item "$S\doc_extract\corrections.py" doc_extract\
   Copy-Item "$S\tests\test_corrections.py" tests\

2. Run the tests:

   python -m pytest -q

   The last line is 98 passed. Your project now matches the snapshot end-of-m05 file for file, except README.md and .env.example: the snapshot’s versions list this module’s commands and the setting REVIEW_DB. Copy them from the snapshot if you want them.

The database is the file out/review.db. The setting REVIEW_DB can name another file. Like everything in out/, you can delete it to start again.

Corrections in pen

An accountant keeps the books in pen. When a line is wrong, the accountant does not erase it or paint over it. They write a new line that corrects the old one, with the date and their initials. Months later, an auditor can see the first line, the correction, who made it and when. The books tell the whole story, including the mistakes.

A review queue needs the same habit. A reviewer changes values every day. Later, someone asks: “What did the pipeline read? Who changed it? Why?” The answers must still be there.

The problem: a correction that overwrites

The easy way to correct a value is to replace it. Grace checks the digital receipt D011, compares it with the customer’s paper, and types the total 211.86 over the stored 211.68. The result looks clean: one value per field.

Now three things are lost. First, what the pipeline read: the next person cannot see that the reader said 211.68, with a box on the page that shows 211.68. Second, who changed it and why. Third, the way back. And in this story Grace swapped two digits: the paper says 211.68, so the pipeline was right and the correction is wrong. Without the original, nobody can find that out, except by asking for the paper again.

Overwriting also breaks the measurements of the next module. If reviewers overwrite the extracted values, the pipeline looks better than it is.

The idea: append-only records

The project keeps the extraction and the corrections in two tables, and never changes a row in either. A change is always a new row. This design is called append-only: you can append rows at the end, and you cannot update or overwrite one. The rows together are the version history of a field. Some teams call it an audit trail.

• The table extractions keeps every field as the pipeline extracted it. A row has the value, its status and reason, its evidence (page and box), and the versions of the pipeline and the contract.
• The table corrections keeps every change by a person. A row has the field, the value before, the new value, the reviewer, a note and a version number.
• The current value of a field is its latest correction, or the extracted value when nobody corrected it.

Two rules protect this. The database refuses an UPDATE on either table: a trigger, a small rule stored in the database, stops it with a message. And the code refuses a correction without a note, because a correction that nobody can explain cannot be checked.

The two tables for D011's total, after the steps below. Each change is a new row; the current value is the latest one. An UPDATE that tries to change the extracted row is refused.

Text description of the diagram

The extracted row keeps D011’s total as the pipeline read it: 211.68, found, on page 1 in the box [14, 227, 213, 236] in points. It also names pipeline 1.0.0 and contract receipt-v1. The first correction row changes it to 211.86, by Grace, version 1, with the note “Typed from the paper receipt”. The second correction row changes it back to 211.68, by Priya, version 2, because version 1 swapped two digits. The current value is 211.68, corrected by Priya. An UPDATE on the extractions table is refused by a trigger.

Correct a receipt, step by step

This is the tiny example: one field of one receipt, with every row visible.

1. Store what the pipeline extracted from D011:

   python -m doc_extract.corrections load D011

   D011: stored 9 fields from the pipeline

2. Record Grace’s correction, with her name and a note:

   macOS / Linux

   python -m doc_extract.corrections correct D011 total 211.86 --reviewer Grace --note "Typed from the paper receipt"

   Windows (PowerShell)

   python -m doc_extract.corrections correct D011 total 211.86 --reviewer Grace --note "Typed from the paper receipt"

   D011 total: correction version 1 recorded; the extracted value is kept

3. Print the history of the receipt:

   python -m doc_extract.corrections history D011

     extracted vendor          Larkfield              found (page 1, box [76, 23, 151, 37])
     extracted order_number    LK-103119              found (page 1, box [14, 102, 101, 111])
     extracted date            2026-08-22             found (page 1, box [14, 89, 213, 98])
     extracted currency        CAD                    found (page 1, box [14, 227, 213, 236])
     extracted subtotal        189.00                 found (page 1, box [14, 189, 213, 198])
     extracted shipping        None                   absent
     extracted tax             22.68                  found (page 1, box [14, 202, 213, 223])
     extracted total           211.68                 found (page 1, box [14, 227, 213, 236])
     extracted payment_method  Cash                   found (page 1, box [14, 252, 213, 261])
     corrected total           211.68 -> 211.86       by Grace, version 1: Typed from the paper receipt

   The nine extracted fields are still there, each with its page and box. The last row is the correction: the old value, the new value, who, the version and the note.

4. Print the current values:

   python -m doc_extract.corrections current D011

     vendor          Larkfield      extracted (found)
     order_number    LK-103119      extracted (found)
     date            2026-08-22     extracted (found)
     currency        CAD            extracted (found)
     subtotal        189.00         extracted (found)
     shipping        None           extracted (absent)
     tax             22.68          extracted (found)
     total           211.86         corrected by Grace (version 1)
     payment_method  Cash           extracted (found)

   The total is now Grace’s 211.86, and the line says where it comes from. The other fields still come from the extraction.

Now try to break the rules. In python, try to change the extracted row in place, then look at both tables:

import sqlite3
from doc_extract import corrections
db = corrections.connect()
try:
    db.execute("UPDATE extractions SET value = '211.86' WHERE doc_id = 'D011' AND field = 'total'")
except sqlite3.IntegrityError as e:
    print("refused:", e)
for row in db.execute("SELECT id, value, status FROM extractions WHERE doc_id = 'D011' AND field = 'total'"):
    print("extractions:", row)
for row in db.execute("SELECT id, old_value, new_value, reviewer, note, version FROM corrections WHERE doc_id = 'D011'"):
    print("corrections:", row)

refused: an extraction is never changed: add a correction
extractions: (8, '211.68', 'found')
corrections: (1, '211.68', '211.86', 'Grace', 'Typed from the paper receipt', 1)

The trigger stops the UPDATE with its message, and the extracted row still says 211.68. The correction is a separate row with its own ID. Then try a correction without a reason:

from doc_extract import corrections
db = corrections.connect()
try:
    corrections.correct(db, "D011", "total", "211.68", "Priya", "   ")
except ValueError as e:
    print("refused:", e)

refused: a correction needs a note that says why

Now Priya checks the paper again. It says 211.68: Grace swapped two digits. Priya does not delete Grace’s row. She adds a second correction:

python -m doc_extract.corrections correct D011 total 211.68 --reviewer Priya --note "The paper receipt says 211.68; version 1 swapped two digits"
python -m doc_extract.corrections history D011
python -m doc_extract.corrections current D011

D011 total: correction version 2 recorded; the extracted value is kept
  ...
  extracted payment_method  Cash                   found (page 1, box [14, 252, 213, 261])
  corrected total           211.68 -> 211.86       by Grace, version 1: Typed from the paper receipt
  corrected total           211.86 -> 211.68       by Priya, version 2: The paper receipt says 211.68; version 1 swapped two digits
  ...
  total           211.68         corrected by Priya (version 2)

The history now tells the whole story: the pipeline read 211.68, Grace changed it to 211.86, and Priya changed it back, with a reason. The current total is version 2. Even a wrong correction is evidence: it shows where the review process can fail, for example a total typed by hand.

Larkfield: a cut-off photo and the customer’s paper

The photo D045 is in the queue because its total is missing. Grace asks the customer for the paper receipt, and the customer sends it. It shows TOTAL CAD 405.61 and Cash 405.61, the lines below the cut edge. Grace records three corrections:

1. Store the extraction and look at its current values:

   python -m doc_extract.corrections load D045
   python -m doc_extract.corrections current D045

   D045: stored 9 fields from the pipeline
     vendor          Larkfield      extracted (found)
     order_number    LK-754892      extracted (found)
     date            2026-06-13     extracted (found)
     currency        CAD            extracted (unverified)
     subtotal        358.95         extracted (found)
     shipping        None           extracted (absent)
     tax             46.66          extracted (found)
     total           None           extracted (absent)
     payment_method  None           extracted (absent)

   The currency is unverified: the model gave CAD, and the photo does not print it. The total and the payment are absent.

2. Record the three values from the paper, each with its note:

   python -m doc_extract.corrections correct D045 total 405.61 --reviewer Grace --note "From the paper receipt the customer sent: TOTAL CAD 405.61"
   python -m doc_extract.corrections correct D045 payment_method Cash --reviewer Grace --note "From the paper receipt the customer sent: Cash 405.61"
   python -m doc_extract.corrections correct D045 currency CAD --reviewer Grace --note "The paper receipt prints TOTAL CAD; the photo cut it off"

   The currency keeps the same value, CAD. The correction still matters: it records that a person checked it, on which paper, and so the value is no longer only the model’s.

3. Check the current values again with the rules of the first lesson. In python:

   from doc_extract import corrections, validate
   db = corrections.connect()
   values = {name: v["value"] for name, v in corrections.current(db, "D045").items()}
   problems = validate.validate(values)
   print(values)
   print([(p.severity, p.field, p.rule) for p in problems] or "no problem")

   {'vendor': 'Larkfield', 'order_number': 'LK-754892', 'date': '2026-06-13', 'currency': 'CAD', 'subtotal': '358.95', 'shipping': None, 'tax': '46.66', 'total': '405.61', 'payment_method': 'Cash'}
   no problem

   358.95 + 46.66 = 405.61, so the corrected receipt passes every check. In the life of a receipt, this is the step “the checks run again and pass” before it goes to the agents.

D045 after review. The extracted rows keep what the photo showed: nothing for the total and the payment. The correction rows add what the paper shows, with who and why.

Text description of the diagram

The photo D045 is cut off below the tax line. The extracted rows keep what the pipeline found on it: the currency CAD, unverified, and nothing for the total and the payment. The correction rows add the total 405.61, the payment Cash and a confirmed currency CAD. Each is by Grace, version 1, with a note that names the paper receipt. The current values are the corrections: currency CAD, total 405.61, payment Cash.

Look at the history of D045 yourself:

python -m doc_extract.corrections history D045

  extracted vendor          Larkfield              found (page 1, box [428, 531, 769, 588])
  extracted order_number    LK-754892              found (page 1, box [163, 854, 544, 901])
  extracted date            2026-06-13             found (page 1, box [163, 803, 1026, 863])
  extracted currency        CAD                    unverified
  extracted subtotal        358.95                 found (page 1, box [172, 1509, 1002, 1561])
  extracted shipping        None                   absent
  extracted tax             46.66                  found (page 1, box [170, 1557, 1002, 1599])
  extracted total           None                   absent
  extracted payment_method  None                   absent
  corrected total           None -> 405.61         by Grace, version 1: From the paper receipt the customer sent: TOTAL CAD 405.61
  corrected payment_method  None -> Cash           by Grace, version 1: From the paper receipt the customer sent: Cash 405.61
  corrected currency        CAD -> CAD             by Grace, version 1: The paper receipt prints TOTAL CAD; the photo cut it off

The extracted rows keep the boxes on the photo. The corrections have no box, because they come from paper, not from the file: their evidence is the note. If someone asks in a year where 405.61 came from, the answer is in the row.

Deletion is a different operation

The triggers refuse UPDATE, not DELETE. This is on purpose. A customer can ask Larkfield to delete their data, and then the pipeline must remove the rows of that document, its extraction and its corrections. That is a deliberate operation, written to a log, which the next module builds (pipeline delete). A correction is never a deletion: it adds a row, and the rows before it stay.

When to use append-only corrections, and what they cost

• Use them whenever a person changes a value that a system extracted: receipts, invoices, forms, evaluation labels.
• Ask for a reason every time. A short note such as “From the paper receipt” is enough. It turns a change into evidence.
• What it costs: more rows, and every reader of the data must ask for the current value, not read the extraction directly. corrections.current does that in one place.
• A limit of this design: if the pipeline reads a document again, the new extraction is a new row. An old correction still wins over it. The next module plans reprocessing.

Common mistake

Fix it in the database

Symptom: A developer sees a wrong total in the review database and runs a quick fix: UPDATE extractions SET value = '211.86' WHERE .... The database answers an extraction is never changed: add a correction. The developer is annoyed and asks to remove the trigger “for a minute”.

Cause: The trigger does its job. Without it, the fix would overwrite the pipeline’s value. Nobody could later see what the pipeline read, who changed it or why. If the fix is wrong, as Grace’s 211.86 was, the right value is lost too.

Diagnose: Look for changes without a history. In this project, a value that differs from its extraction must have a correction row: corrections history <document> shows it. A changed extraction row with no correction is a sign that someone bypassed the workflow.

Fix: Keep the triggers. Make every change through corrections correct, with a reviewer and a note, even a change by a developer. If a bug makes many values wrong, fix the reader and extract again. The new extraction is a new row with a new pipeline version.

Consequence in AI work: the same rule protects labels and evaluation sets. A label changed in place destroys the evidence of how the labels were made. Keep the old label and add the new one with a reason, as Label and version in Evaluating and Testing AI Systems does.

Your turn: the photo cut off at the top

Change one condition: D047, whose top is cut off, so the vendor and the date are missing. The customer sends the paper receipt. Its first lines are, from the course’s generator:

LARKFIELD
Home & Garden
88 Queen St W
Toronto ON M5H 2M5
------------------------------------
Date: 2026-08-01 10:39

Camille handles this ticket. Store the extraction, record the two corrections with Camille’s name and a note, print the history, and check the current values with validate. Predict the version numbers before you run.

Show the answer

python -m doc_extract.corrections load D047
python -m doc_extract.corrections correct D047 vendor Larkfield --reviewer Camille --note "Paper receipt from the customer: LARKFIELD at the top"
python -m doc_extract.corrections correct D047 date 2026-08-01 --reviewer Camille --note "Paper receipt from the customer: Date: 2026-08-01"
python -m doc_extract.corrections history D047

D047: stored 9 fields from the pipeline
D047 vendor: correction version 1 recorded; the extracted value is kept
D047 date: correction version 1 recorded; the extracted value is kept
  extracted vendor          None                   absent
  extracted order_number    LK-192531              found (page 1, box [124, 26, 512, 75])
  extracted date            None                   absent
  extracted currency        CAD                    found (page 1, box [158, 610, 1018, 674])
  extracted subtotal        187.00                 found (page 1, box [151, 501, 1019, 568])
  extracted shipping        None                   absent
  extracted tax             24.31                  found (page 1, box [154, 557, 1019, 621])
  extracted total           211.31                 found (page 1, box [158, 610, 1018, 674])
  extracted payment_method  Visa                   found (page 1, box [162, 714, 1023, 779])
  corrected vendor          None -> Larkfield      by Camille, version 1: Paper receipt from the customer: LARKFIELD at the top
  corrected date            None -> 2026-08-01     by Camille, version 1: Paper receipt from the customer: Date: 2026-08-01

In python, the check of the current values:

from doc_extract import corrections, validate
db = corrections.connect()
values = {name: v["value"] for name, v in corrections.current(db, "D047").items()}
print([(p.severity, p.field, p.rule) for p in validate.validate(values)] or "no problem")

no problem

Both corrections are version 1, because the version counts the corrections of one field, not of the whole receipt. The extracted rows keep “absent” for the vendor and the date: that is what the photo shows. The two required fields now have current values, so no required check fails. The receipt has no error left.

Knowledge check

Check your understanding

Explain what the history of D011 lets an auditor find out that an overwrite would hide.

Show the answer

The history shows the extracted total 211.68 with its box on page 1. So the auditor sees that the pipeline read the paper correctly. It shows Grace’s correction to 211.86 with her note, and Priya’s correction back to 211.68 with the reason: version 1 swapped two digits. An overwrite would keep only the last value. The auditor could not see that the pipeline was right, that a correction was wrong, or who made each change and why.

Check your understanding

Predict: Priya runs corrections correct D011 total 211.68 again with another note. What is stored, and what is the current value?

Show the answer

A third correction row is stored for D011’s total: version 3, old value 211.68, new value 211.68, by Priya, with the new note. No row is changed or removed. The current value stays 211.68, now from version 3. A correction with the same value is still a record: it says that a person checked the field again, when and why.

Check your understanding

Practical check: run python -m doc_extract.corrections correct D011 colour red --reviewer Sam --note 'a test'. What happens, and why is that right?

Show the answer

The command stops with ValueError: colour is not a field of the contract, and nothing is stored. The contract has nine fields, and a correction can change only one of them. A correction to an unknown field adds a value that no other part of the pipeline understands. It also hides a typing mistake in the field name.

Module recap

This module made the pipeline’s results safe to use. In Business validation, you checked the values against the rules of the business. Five receipt documents have an error, including D050, whose nine values are read right and whose total is ten dollars too high. In Confidence and review routing, you measured that the model’s own confidence is not calibrated. You routed by reasons and by a measured certainty: 11 of 54 documents to review, and no wrong total to the agents. In this lesson, a reviewer’s correction became a new row with a name, a note and a version, next to the extraction that stays. Your project matches the snapshot end-of-m05: python -m pytest -q prints 98 passed.

Module practice: a review queue for failed totals and uncertain vendor names

Design the review queue for two kinds of problem: totals that fail validation, and vendor names that the pipeline is not sure of. A design says what goes into the queue, what the reviewer sees, and what the reviewer can do.

Guided: failed totals

1. List the documents with a failed total. They have the error adds_up or a missing total:

   python -m doc_extract.review

   In the output, D050 and D051 have a total that does not add up, and D045 has a missing total. D046’s line is long, and the command cuts it: validate --all shows its missing total too. Each line gives the reason in words.

2. Write what the reviewer sees for a failed total. Use the queue entry of D051 from Confidence and review routing, and the picture of D050 and D051 from Business validation. The course’s design, from docs/review-queue.md:

   One card per document: the picture of the page with the box of every field drawn on it,
   the extracted values with their status (found, inferred, absent and why), the reasons,
   and for a conflict both values side by side.

   For a failed total, the boxes that matter are the subtotal, the tax and the total. The reason gives the sum: 528.00 + 0 + 68.64 = 596.64, but the total is 595.74.

3. Write what the reviewer can do, and how each action is stored:

   • Accept: the receipt really prints a total that does not add up. The reviewer accepts it and writes why. The agent handles the customer, for example with the amount on the bank statement. Stored: a correction with the same value and a note.
   • Correct: the paper shows another value. Stored: a correction with the new value and a note, as you did for D045.
   • Ask the customer: the page does not show the value, as on D045 and D046. Nothing is stored until the customer sends a better picture.

4. Check the design against the course’s measurement: at the threshold 0.90, the queue has 11 documents, and every failed total is in it.

Independent: uncertain vendor names

Now do the same for vendor names. A vendor name is uncertain when it is not one of the shops that Larkfield knows, in contract.VENDORS. It can be a misread such as LARKEFIELD, or a new partner shop. Decide:

1. Is an unknown vendor an error or a warning? Why?
2. What does the reviewer see, and what can they do?
3. Which documents of the course would go to the queue? Write a check and run it on the pipeline’s values and on the model alone.

Show a worked solution

1. An error. An unknown vendor is not a normal case at Larkfield, as a US receipt is. Either the name is misread, or the receipt is from a shop that Larkfield does not accept. Both need a person before a refund.

2. The card and the actions. The reviewer sees the vendor’s box on the page, the extracted name, and the list of known shops. They can correct the name to a known shop, with a note (“printed LARKFIELD; read LARKEFIELD”). If the shop is really new, they ask the purchasing team to add it to the list, which is a change to the contract. Until then, the receipt waits.

3. The check. In python, the vendor check on the pipeline and on the model alone:

from doc_extract import combine, contract, documents, vision
for d in documents.receipts():
    ex, _ = combine.combine(d)
    model = vision.read(d.file, d.truth.get("receipt_page", 1))
    for who, value in (("pipeline", ex.value("vendor")), ("model alone", model.value("vendor"))):
        if value is not None and contract.vendor(value) not in contract.VENDORS:
            print(d.doc_id, who, repr(value), "is not in the vendor list")
print("checked", len(documents.receipts()), "receipt documents")

D009 model alone 'LARKEFIELD' is not in the vendor list
checked 54 receipt documents

The pipeline gives no unknown vendor on the course’s documents: the rules read D009 from the text layer. The model alone would put D009 in the queue, with its misread LARKEFIELD. A misread like this breaks no other rule, so this check closes one of the holes of validation from the first lesson. To make it part of the project, add it to validate.validate as an error, with a test. The course’s docs/review-queue.md describes the same design.

Completion check: corrections without erasing evidence

Explain how a correction is recorded without erasing the original evidence. Use D045 as the example. Say where the extracted value is kept, where the correction is kept, and how the pipeline knows the current value.

Show a worked solution

• The extracted value stays in the table extractions, as the pipeline wrote it. For D045, the total is absent, and the found values keep their boxes on the photo. A trigger refuses any UPDATE of this table, so nobody can change the row, not even by mistake.
• The correction is a new row in the table corrections: field total, None -> 405.61, Grace, version 1, with a note that names the paper receipt. A correction without a note is refused. The correction table refuses UPDATE too, so a wrong correction is fixed with a new version, as Priya did on D011.
• The current value is the latest correction of the field, or the extracted value if there is none. corrections current D045 shows total 405.61 corrected by Grace (version 1), and corrections history D045 shows both rows.

So the evidence of the photo and of the paper are both kept, each with its source. A later reader can follow every change back to a person and a reason.

Recap

• Never overwrite an extracted value. Keep it, and add each change as a new row with who, why and a version.
• The project’s two tables, extractions and corrections, are append-only: triggers refuse UPDATE, and a correction needs a note.
• The current value is the latest correction, or the extraction. A wrong correction is fixed by a new version, and the history keeps both.
• Corrected values are checked again before they go to the agents: D045 passes every check with the total from the paper.
• Deletion is a separate, deliberate operation with a log, not a correction.

Next, Module 6 measures the pipeline field by field, against the truth, by document type.

Next: Field-level metrics