Integration and maintenance

What you will be able to do

Run the pipeline next to the ticket system without making anyone wait. Keep track of which version made each result, run old results again after a change, and delete a customer’s data on request. Check every change against a fixed set of documents before it goes live.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The review queue and the corrections, from Confidence and review routing and Correction workflow.
• The warning 3 x 74.85 is not 74.85, from Business validation. This lesson fixes the bug behind it.
• A job queue with workers, from Job lifecycle in Scaling APIs and AI Workloads, if you took it. This lesson explains the idea in a few paragraphs.

What you need on your computer

• Your project at the end of the previous lesson: python -m pytest -q prints 103 passed.

Tested with Python 3.12.13 and RapidOCR 3.10.0 on macOS (Apple silicon). The Windows and Linux commands are not tested.

Get the lesson’s files

1. Copy the pipeline, its test, the regression set and two notes for the docs/ folder:

   macOS / Linux

   S=../nextia-learning-labs/C13/snapshots/end-of-m06
   cp $S/doc_extract/pipeline.py doc_extract/
   cp $S/tests/test_pipeline.py tests/
   mkdir -p tests/regression
   cp $S/tests/regression/expected.json tests/regression/
   cp $S/docs/operations.md $S/docs/evaluation-report.md docs/

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C13\snapshots\end-of-m06"
   Copy-Item "$S\doc_extract\pipeline.py" doc_extract\
   Copy-Item "$S\tests\test_pipeline.py" tests\
   New-Item -ItemType Directory -Force tests\regression | Out-Null
   Copy-Item "$S\tests\regression\expected.json" tests\regression\
   Copy-Item "$S\docs\operations.md", "$S\docs\evaluation-report.md" docs\

   Do not copy fields.py yet. You change it yourself in this lesson.

2. Run the tests:

   python -m pytest -q

   The last line is 107 passed.

Leave your shirt at the cleaner’s

You take a shirt to a dry cleaner. You do not stand at the counter while they clean it. They give you a ticket at once, and you come back later. The ticket says what they did: the date and the process. When the cleaner buys a new machine, they first clean a few test shirts, one of each fabric, and compare them with the old results. Only then do they use it for everybody’s clothes. And when you ask them to forget you, they remove your name from their books.

Each of these habits solves a problem that the pipeline also has: waiting, change, testing and deletion.

The problem: a pipeline inside a busy system

Amira connects the pipeline to ticket-api, Larkfield’s ticket system. Her first idea is simple: when a customer attaches a receipt, the ticket system calls the pipeline and waits for the values. On a digital receipt, that takes milliseconds. On a photo, OCR takes about a second. When the model is needed, the median call takes 7 seconds, and one recorded call took 132 seconds. The customer’s upload waits for all of it, and a web request that waits two minutes fails.

Then the pipeline changes. In Business validation, the item check found a bug in the rules. After a fix, which stored results are old? Which still carry the old warning? Nobody knows, because a result does not say which code made it.

Two more questions come from Grace and from the law. A customer asks Larkfield to delete their data: what must go? And how does the team know that a fix for one bug does not break another document type?

The idea: background jobs, versioned results, a regression set

Four habits, as at the cleaner’s:

• Background processing. The ticket system stores the file and puts a job in a queue: “read attachment 7 of ticket 1234”. It answers the customer at once. A separate program, a worker, takes the next job, runs the pipeline and stores the result. Nobody waits for the model.
• Versioned results. Every result stores the versions that made it: the pipeline, the contract, the OCR engine with its settings, and the model prompt. After a change, you find the old results and reprocess them: you run them again with the new code.
• Deletion. One command deletes everything that the pipeline made from one document, and writes a line in a log: what and when, never the content.
• A regression set. A regression is a change that makes something worse that worked before. The regression set is a fixed list of documents, at least one of each type, with their expected results. Every change must give the same results, or explain each difference.

The pipeline as a background worker next to the ticket system. The customer and the agent never wait for OCR or the model.

Text description of the diagram

A sequence diagram. The customer attaches a receipt to a ticket. The ticket system stores the original file, puts a job in the queue, and answers the customer at once. An agent who opens the ticket sees “reading the receipt”. The worker takes the next job and runs intake, reading, extraction, validation and the review decision. This takes from 5 milliseconds with the text layer to about 10 seconds with the model. The worker sends the result, with the values, their boxes and the versions, to the ticket system. When the agent opens the ticket again, they see the values with their boxes, or “needs review”.

The course project has no queue, because the queue belongs to the ticket system. Job lifecycle builds one. The project gives the worker its one function: pipeline.process reads one document and returns one result. docs/operations.md describes how the two parts fit together.

Run the whole pipeline

1. Run every document through the pipeline:

   python -m doc_extract.pipeline run

   65 documents: straight to the agents 48, to review 12, stopped at intake 5
   results in out/results/

   The 48 include the four documents that are not receipts: the two policies, the price list and the spreadsheet. The 12 for review are the 11 of the previous module and D060, a second scan of D020. The 5 stopped are the exact copy, the damaged file, the locked file, the file that is too large and the text file.

2. Look at one result. In python:

   import json
   r = json.load(open("out/results/D011.json", encoding="utf-8"))
   print("versions:", r["versions"])
   print("intake:  ", r["intake"]["status"])
   total = r["fields"]["total"]
   print("total:   ", total["value"], total["status"], "page", total["evidence"]["page"],
         total["evidence"]["box"], total["evidence"]["unit"], repr(total["evidence"]["seen"]))
   print("review:  ", r["review"])

   versions: {'pipeline': '1.0.0', 'contract': 'receipt-v1', 'ocr': 'rapidocr 3.10.0, 200 dpi, max side 1280', 'vision': 'recorded chat-strong, prompt v2'}
   intake:   accepted
   total:    211.68 found page 1 [14.2, 226.8, 212.6, 235.8] pt 'TOTAL  CAD 211.68'
   review:   {'needed': False, 'reasons': []}

   One file holds everything that an agent and an auditor need. It has the value, its box on the page and the text seen there. It also has the review decision and the four versions.

3. Ask which results are old:

   python -m doc_extract.pipeline status

   65 results; 0 made by an older version: -
   current versions: {'pipeline': '1.0.0', 'contract': 'receipt-v1', 'ocr': 'rapidocr 3.10.0, 200 dpi, max side 1280', 'vision': 'recorded chat-strong, prompt v2'}

4. Run the regression set:

   python -m doc_extract.pipeline regression

   regression set: 15 documents, no change

The regression set has 15 documents, at least one of each kind that the course knows: a digital receipt (D001), a French one (D004), the other shop (D017), a French scan (D021), a turned scan (D025), a French photo (D036), the photo cut off at the bottom (D045) and at the top (D047), the unreadable photo (D046), a partner photo (D049), the wrong total (D050), the claim with two parts (D058), the exact copy (D059), the damaged file (D061) and the JPEG named .pdf (D064). For each one, tests/regression/expected.json stores the intake decision, every field’s value and status, and the review decision.

Fix a bug the safe way

Now use these tools on a real bug. In the previous module, the item check warned about many receipts. Count the stored results that have this warning. In python:

import json
from pathlib import Path
hit = [p.stem for p in sorted(Path("out/results").glob("D*.json"))
       if any(v["rule"] == "line" for v in json.loads(p.read_text(encoding="utf-8")).get("validation", []))]
print(len(hit), "results with a 'line' warning:", ", ".join(hit) or "-")

16 results with a 'line' warning: D001, D003, D005, D007, D009, D013, D015, D022, D027, D028, D031, D036, D039, D043, D047, D058

python -m doc_extract.validate D001

D001  receipt-lk-986875.pdf:
  warning items          line      3 x 74.85 is not 74.85 (Pruning shears)

The bug on one item line of D001. The receipt is right; the reader took two amounts as one. The boxes come from the text layer.

Text description of the diagram

Part of the digital receipt D001, with boxes from the text layer: the item name Pruning shears, the line 3 x 24.95, and the amount 74.85 on the same row. Before the fix, the pattern for a quantity line takes “24.95 74.85” as one amount, and the money reader keeps the last number. So the unit price is 74.85, and 3 × 74.85 = 224.55 is not the line total 74.85: a warning. After the fix, the pattern takes one amount, 24.95. Then 3 × 24.95 = 74.85, and the item check passes.

The receipt is right, and the check is right. The reader is wrong. A careful fix has four steps: a test that fails, the fix, a new version, and the checks.

1. Write a test that shows the bug. Add this test at the end of tests/test_fields.py:

   def test_a_quantity_line_keeps_the_unit_price():
       # Found by the item check of validate.py: "3 x 24.95  74.85" gave a unit price of 74.85.
       rows = [(1, [Line("Pruning shears", (14, 227, 90, 236))]),
               (1, [Line("3 x 24.95", (25, 239, 74, 248)), Line("74.85", (186, 239, 213, 248))]),
               (1, [Line("Tuyau d'arrosage", (14, 252, 90, 261))]),
               (1, [Line("2 x 1 011,78", (25, 264, 74, 273)), Line("2 023,56", (180, 264, 213, 273))])]
       got = [(i["quantity"], i["unit_price"], i["line_total"]) for i in fields.items(rows)]
       assert got == [(3, "24.95", "74.85"), (2, "1011.78", "2023.56")]

   The test builds two item lines by hand, with boxes. The first is D001’s line. The second is a French line with a space in the thousands, which the old pattern was written for. The file already imports fields and Line.

   python -m pytest -q tests/test_fields.py

   E       AssertionError: assert [(3, '74.85',...', '2023.56')] == [(3, '24.95',...', '2023.56')]
   E
   E         At index 0 diff: (3, '74.85', '74.85') != (3, '24.95', '74.85')
   E         Use -v to get more diff
   
   <project>/tests/test_fields.py:58: AssertionError
   =========================== short test summary info ============================
   FAILED tests/test_fields.py::test_a_quantity_line_keeps_the_unit_price - Asse...
   1 failed, 5 passed in 5.36s

   The test fails for the right reason: the unit price is 74.85. <project> stands for your project folder. A test that fails first proves that it can see the bug.

2. Fix the reader. In doc_extract/fields.py, change the line that defines QTY_LINE:

   QTY_LINE = re.compile(r"^(\d{1,2})\s*[x×X*]\s*\$?\s*(\d[\d ,.]*)")
   # "3 x 24.95  74.85": group 2 is the unit price, one amount ("1 011,78" in French). The second
   # choice keeps a line whose price OCR misread (no decimals) as an item, as before.
   QTY_LINE = re.compile(r"^(\d{1,2})\s*[x×X*]\s*\$?\s*(\d{1,3}(?:[ ,.]\d{3})*[.,]\d{2}(?!\d)|\d[\d ,.]*)")

   The old pattern took any run of digits, spaces, commas and points. The new one first tries one amount: one to three digits, groups of three digits, then two decimals. If OCR misread the price and that fails, the old choice still applies, so no item line is lost. If you prefer, copy doc_extract/fields.py from the end-of-m06 snapshot: it is the same change.

   python -m pytest -q tests/test_fields.py
   python -m pytest -q

   6 passed in 5.34s
   108 passed in 17.26s

   python -m doc_extract.validate D001

   D001  receipt-lk-986875.pdf: no problem

3. Ask which results are old:

   python -m doc_extract.pipeline status

   65 results; 0 made by an older version: -
   current versions: {'pipeline': '1.0.0', 'contract': 'receipt-v1', 'ocr': 'rapidocr 3.10.0, 200 dpi, max side 1280', 'vision': 'recorded chat-strong, prompt v2'}

   Zero. The code changed, but the version did not, so the pipeline cannot tell the old results from new ones. The 16 stored warnings stay. Give the change a version: in doc_extract/__init__.py, write __version__ = "1.0.1", and in pyproject.toml, write version = "1.0.1". Then ask again:

   python -m doc_extract.pipeline status

   65 results; 65 made by an older version: D001, D002, D003, D004, D005, D006, D007, D008, D009, D010, D011, D012, D013, D014, D015, D016, D017, D018, D019, D020, D021, D022, D023, D024, D025, D026, D027, D028, D029, D030, D031, D032, D033, D034, D035, D036, D037, D038, D039, D040, D041, D042, D043, D044, D045, D046, D047, D048, D049, D050, D051, D052, D053, D054, D055, D056, D057, D058, D059, D060, D061, D062, D063, D064, D065
   current versions: {'pipeline': '1.0.1', 'contract': 'receipt-v1', 'ocr': 'rapidocr 3.10.0, 200 dpi, max side 1280', 'vision': 'recorded chat-strong, prompt v2'}

4. Reprocess, then run the regression set:

   python -m doc_extract.pipeline reprocess
   python -m doc_extract.pipeline status
   python -m doc_extract.pipeline regression

   reprocessed 65 documents
   65 results; 0 made by an older version: -
   current versions: {'pipeline': '1.0.1', 'contract': 'receipt-v1', 'ocr': 'rapidocr 3.10.0, 200 dpi, max side 1280', 'vision': 'recorded chat-strong, prompt v2'}
   regression set: 15 documents, no change

   Run the count of line warnings from the start of this section again:

   0 results with a 'line' warning: -

The regression set shows no change, and that is correct. It compares the fields, their statuses and the review decisions. The fix changes none of them, because the item lines are not one of the nine fields. So the regression set did not see the bug, and it does not see the fix either. The new unit test guards the fix. This is why a bug fix starts with a test. The regression set checks that nothing else moved.

The order of a safe change, as the course fixed its bug. Each step has a command and an output you can check.

Text description of the diagram

The order of a safe change. A warning points at the reader. You write a test that fails, then fix the reader and run all the tests. You give the change a new version, so that the status command finds the old results, and you reprocess them. Then you run the regression set. With no change, you release. With a change, you explain each change, or you fix the code and start again with a test.

Delete a customer’s data

Grace gets a request: the customer of D045 wants Larkfield to delete their data. In the previous module, a reviewer stored a correction for D045. Store one now, if your database has none, so that you can see what deletion removes:

python -m doc_extract.corrections load D045
python -m doc_extract.corrections correct D045 total 405.61 --reviewer Grace --note "From the paper receipt the customer sent: TOTAL CAD 405.61"

Then delete:

python -m doc_extract.pipeline delete D045

  removed out/results/D045.json
  removed out/cache/ocr/6e73e737a59b3abb-p1-200-1280-raw-rapidocr-3.10.0.json
  removed out/cache/ocr/6e73e737a59b3abb-p1-200-1280-prep-rapidocr-3.10.0.json
  removed out/review.db: 9 extraction rows, 1 correction rows
The original file belongs to the ticket system: delete it there (this pipeline never changes it).

If you kept the database of the previous module, D045 has 3 correction rows there, not 1. The pipeline deletes what it made: the result, the OCR cache of both pictures, and the rows of the review database, also the correction. Deletion is the one operation that removes corrections, on purpose and with a log. The log is the file out/deletions.log. Its line says what and when, never the values. Your line has the time of your own run:

{"doc_id": "D045", "deleted_at": "<the date and time of your run, in UTC>", "removed": 4}

The original photo is not deleted. It belongs to the ticket system, which deletes it with the ticket. In the course project, the photo stays in data/documents/, so a later pipeline run reads it again. In production, the ticket system deletes the file first, and then nothing can bring the data back.

When to use each habit, and what it costs

• Background processing pays when any path can be slow, as soon as OCR or a model is in the pipeline. It costs a queue, a worker to run and watch, and a “reading…” state in the agent’s screen.
• Versions cost one line in each result. Change the version with every change to the code, the contract, the OCR settings or the prompt, or status cannot help you.
• Reprocessing costs the same as the first run: OCR time and model calls. With the cache, the course’s 65 documents take about a second. A new OCR setting reads every picture again, and a new prompt calls the model again for every page that needs it. Budget it with the previous lesson.
• Corrections survive reprocessing. A person’s correction is the latest row, so a new extraction never replaces it, as docs/operations.md says.
• A regression set costs labels and a few seconds per change. Keep at least one document of each type and each failure you have seen. Add a document each time you fix a bug that it would have shown.

Common mistake

The fix that nobody reprocessed

Symptom: A developer fixes the unit-price bug and ships it. A week later, Grace’s team still sees 3 x 74.85 is not 74.85 on old receipts. New receipts are fine. The developer says: “That is impossible, the code is fixed.”

Cause: The code changed, but the version did not. pipeline status compares the stored versions with the current ones, so it found nothing old. The stored results kept the old warnings. You saw this in step 3: 0 made by an older version right after the fix. The week of complaints is a constructed story.

Diagnose: Look at the stored versions and the code’s version: python -m doc_extract.pipeline status. If they are equal, but a stored result shows a problem that the current code does not give, the version was not changed.

Fix: Change the version with the code, in the same commit. Then run status, reprocess and regression.

Consequence in AI work: stored outputs outlive the code that made them. Embeddings, labels, summaries and extractions all need the version of the model, prompt and code that made them. Without it, you cannot find the outputs to refresh.

Your turn: a smaller picture for OCR

Change one condition: the OCR setting. With OCR_MAX_SIDE=960, OCR reads smaller pictures, and a worker needs less memory. Predict: does the regression set change? Then run it with the new setting:

macOS / Linux

OCR_MAX_SIDE=960 python -m doc_extract.pipeline regression

Windows (PowerShell)

$env:OCR_MAX_SIDE = "960"
python -m doc_extract.pipeline regression
Remove-Item Env:OCR_MAX_SIDE

The command reads every picture again with the new setting, so it takes about 40 seconds. Then put the results back with the normal setting:

python -m doc_extract.pipeline status
python -m doc_extract.pipeline reprocess
python -m doc_extract.pipeline regression

Show the answer

D045 currency: expected ['CAD', 'unverified'], got ['CAD', 'inferred']

The command stops with an error (exit code 1), and the regression shows one change. Everything else in the 15 documents stays the same. On the cut-off photo D045, the currency keeps its value, CAD, but not its status. With the normal setting, OCR finds no currency. The model gives CAD, but its value is not printed on the page, so it is “unverified”. With smaller pictures, OCR reads the address line Toronto ON M5H 2M5, the rules infer CAD from it, and the model agrees. An unverified value is a reason for review, and an inferred one is not. D045 goes to review anyway, for its missing total, but on another receipt the same change could decide the route. So the change needs a decision, not a silent release.

The regression run saved all 65 results with the new setting. So status lists 65 results made by an older version, and reprocess runs them again with 1,280 pixels. Then the regression set shows no change again. D045 is back too, because its original photo is still in data/documents/.

Knowledge check

Check your understanding

Explain why the ticket system should put a job in a queue instead of calling the pipeline and waiting.

Show the answer

One document takes from 5 milliseconds for a digital PDF to many seconds with the model. One recorded model call took 132 seconds. A customer’s upload that waits that long fails, and many uploads at once would block the ticket system. With a queue, the ticket system stores the file and answers at once. A worker reads the documents one by one, and the agent sees “reading the receipt” until the result arrives.

Check your understanding

Predict: the team changes the prompt of the vision model from v2 to v3, and forgets nothing. Which results does pipeline status list, and what does reprocessing cost?

Show the answer

The versions include the prompt, so every stored result lists prompt v2 and differs from the current versions. status lists all of them, also the digital receipts that never used the model. Reprocessing then reads them all again. With the cache, the text layer and OCR cost almost nothing. The model is called again only for the documents that need it. In the course, that is 9 of the 54 receipt documents, about US$0.10 at the recorded prices. The regression set then shows which values the new prompt changes.

Check your understanding

Practical check: after the fix and the new version, run python -m doc_extract.pipeline regression. Why does it show no change, although 16 results lost a warning?

Show the answer

It prints regression set: 15 documents, no change. The regression set compares the intake decision, every field’s value and status, and the review decision. The fix changes only the unit price of item lines and the warnings about them. Item lines are not one of the nine fields, and a warning does not change the review decision. So the set cannot see the bug or the fix. The new test in tests/test_fields.py guards the fix, and the regression set confirms that nothing else moved.

Module recap

This module measured the pipeline and made it ready to run. In Field-level metrics, you scored every field of every document with five outcomes. The plain prompt reads 97.5% of the printed values right but completes only 40 of 54 documents, because it invents order numbers. The combined pipeline completes 53. In Operational constraints, you budgeted a day of 300 pages. It costs US$0.91 with a quarter of the pages to the model, and US$3.64 with all of them. One OCR worker needs about 1.5 GB of memory. In this lesson, the pipeline became a background worker with versioned results. You fixed a real bug with a test first, and reprocessed after a new version. You deleted a customer’s data with a log, and checked every change against a regression set. Your project matches the snapshot end-of-m06: python -m pytest -q prints 108 passed. Only README.md differs: the snapshot’s version lists this module’s commands. Copy it from the snapshot if you want it.

Module practice: a prototype plan, sample outputs and an evaluation report

Deliver the three documents that a team needs before a prototype goes to users. They are a plan for how the pipeline runs, sample outputs that show its evidence, and an evaluation report.

Guided: the course’s own delivery

1. Read the evaluation report docs/evaluation-report.md. It is the worked example. It has a table by path, an error analysis by document type, and a section “What this does not show”. Check two of its numbers against your own output of python -m doc_extract.evaluate --systems text,ocr,ocr_prepared,vision,vision_v1,combined.

2. Collect sample outputs for a good, an unreadable and a damaged document: out/results/D011.json, out/results/D046.json and out/results/D061.json. The completion check below explains what each one shows.

3. Read the plan docs/operations.md. It covers background processing, versions and reprocessing, deletion, the regression set, and the bug fix of this lesson. Add the numbers of your budget from the previous lesson: pages per day, model share, cost per day and the number of OCR workers.

Independent: the French receipts

Camille’s team in Montréal handles the French receipts. They ask for their own short report before they use the pipeline. Change one condition: evaluate only the French receipt documents. Write half a page:

1. Which French documents are in the set, and of which types?
2. How do the plain prompt, the precise prompt and the combined pipeline do on them: values right, invented values, complete documents?
3. For how many French documents does the combined pipeline call the model?
4. What does your report not show?

Use out/evaluation.json from the first lesson of this module, and documents.receipts(), whose documents have a language.

Show a worked solution

In python:

import json
from doc_extract import combine, documents
result = json.load(open("out/evaluation.json", encoding="utf-8"))
french = [d for d in documents.receipts() if d.language == "fr"]
print("French receipt documents:", ", ".join(f"{d.doc_id} {d.source}" for d in french))
for system in ("vision_v1", "vision", "combined"):
    rows = [result["systems"][system]["per_doc"][d.doc_id] for d in french]
    outcomes = [o for r in rows for o in r["fields"].values()]
    printed = sum(o in ("correct", "missing", "wrong") for o in outcomes)
    print(f"{system:<10} right {outcomes.count('correct')} of {printed}, missing {outcomes.count('missing')}, "
          f"wrong {outcomes.count('wrong')}, invented {outcomes.count('invented')}, "
          f"complete {sum(r['complete'] for r in rows)} of {len(rows)}")
used = [d.doc_id for d in french if combine.combine(d)[1].used_model]
print("the combined pipeline called the model for:", ", ".join(used) or "none")

French receipt documents: D004 digital_pdf, D009 digital_pdf, D012 digital_pdf, D015 digital_pdf, D021 scan_pdf, D026 scan_image, D031 scan_pdf, D036 photo, D040 photo, D044 photo
vision_v1  right 82 of 84, missing 0, wrong 2, invented 0, complete 8 of 10
vision     right 83 of 84, missing 0, wrong 1, invented 0, complete 9 of 10
combined   right 84 of 84, missing 0, wrong 0, invented 0, complete 10 of 10
the combined pipeline called the model for: none

A short report from these numbers:

1. The documents. Ten of the 54 receipt documents are French: four digital PDFs (D004, D009, D012, D015), three scans (D021, D026, D031) and three photos (D036, D040, D044).
2. The results. The combined pipeline reads all 84 printed values right, and all 10 documents are complete. The plain prompt misreads two dates, on D021 and D036. The precise prompt reads D009’s vendor as LARKEFIELD. Neither prompt invents a value here, because every French receipt prints its order number: there is no empty field to fill.
3. The model. The combined pipeline never calls it for these documents. The rules know Larkfield’s French words, such as Sous-total, TPS, TVQ and Paiement. So the French receipts cost no model money.
4. What it does not show. Ten documents is a small slice. One wrong value moves the share by more than one point, and one document moves the completion by 10 points. Every French document is a Larkfield layout made by the course’s generator. A French receipt from a partner shop would need the model, and nobody has measured that case yet.

Completion check: traceable outputs for good, unreadable and damaged documents

Show, for a good document, an unreadable one and a damaged one, where each output came from, and why the pipeline decided what it decided. Use D011, D046 and D061.

Show a worked solution

In python:

import json
for doc in ("D011", "D046", "D061"):
    r = json.load(open(f"out/results/{doc}.json", encoding="utf-8"))
    print(f"{doc} {r['file']}  intake: {r['intake']['status']} {r['intake']['reason']}".rstrip())
    for name in ("vendor", "total"):
        f = r.get("fields", {}).get(name)
        if f:
            ev = f["evidence"]
            where = f"page {ev['page']}, box {ev['box']} {ev['unit']}" if ev and ev.get("box") else "no box"
            print(f"  {name:<7} {str(f['value']):<10} {f['status']:<7} {f['reason'] or '':<11} {where}")
    print(f"  review: {r['review']['needed']}; {'; '.join(r['review']['reasons'])[:150]}")
    print(f"  made by pipeline {r['versions']['pipeline']}, contract {r['versions']['contract']}")

D011 receipt-lk-103119.pdf  intake: accepted
  vendor  Larkfield  found               page 1, box [75.6, 22.7, 151.2, 36.7] pt
  total   211.68     found               page 1, box [14.2, 226.8, 212.6, 235.8] pt
  review: False;
  made by pipeline 1.0.1, contract receipt-v1
D046 IMG_4887.jpg  intake: accepted
  vendor  None       absent  unreadable  no box
  total   None       absent  unreadable  no box
  review: True; vendor: vendor is missing; date: date is missing; currency: currency is missing; subtotal: subtotal is missing; tax: tax is missing; total: total is m
  made by pipeline 1.0.1, contract receipt-v1
D061 receipt-download.pdf  intake: corrupt EOF marker not found
  review: True; intake: corrupt: EOF marker not found
  made by pipeline 1.0.1, contract receipt-v1

• Good (D011, a digital receipt): every value has its page and its box in points on the original PDF. So an agent can see the line it came from. No reason for review: it goes straight to the agents.
• Unreadable (D046, a blurred photo): the values are absent with the reason unreadable, and there is no box, because nothing was read. Nothing is invented. The review reasons name each missing required field, so a person asks the customer for a better photo.
• Damaged (D061, a download that stopped): intake stops it with the reason EOF marker not found. There are no fields, because nothing was read. The result still exists, with its review decision, so the agent knows why there are no values.

All three results name the pipeline and contract versions that made them. So each output can be traced to a file, a page, a box, a reason and a version.

Recap

• The pipeline runs as a background worker. The ticket system answers at once, and nobody waits for OCR or the model.
• Every result stores the versions that made it. Change the version with the code, then status finds the old results and reprocess runs them again.
• pipeline delete removes the result, the OCR cache and the review rows of a document, and writes a log line without content. The original belongs to the ticket system.
• The regression set has 15 documents, one of each kind. Every change must show no change, or explain each one, as D045’s currency with smaller pictures.
• Fix a bug with a test that fails first. The course’s unit-price fix removed 16 false warnings and left the regression set unchanged.

Next, the case studies run the pipeline on real public documents: labelled receipt photos, and a table from a public statistics report.

Next: Read real receipts