Final assignment

What you will be able to do

Use every module of the course on documents that you did not see in it. You make a small synthetic document set of your own, with its true values. Then you build a pipeline that reads it with evidence for every value. It checks the values and sends the doubtful documents to a person who can correct them. Last, you measure where it fails, by document type, and you say what your measurement cannot show.

Before you startWhat to know and to have readyAbout 180 minutes

• You completed Modules 1 to 6. Each part below names the lessons that it uses.
• You do not need the case studies for this assignment. The certificate asks for at least one of them.
• The project: your doc-extract at the end of Module 6, or a copy of the snapshot end-of-m06 (see the snapshots). Work in a new copy, so that the course’s project stays as it is. You may reuse any of its code. Say in your report what you reused and what you changed.
• No account and no money are needed. The vision model is optional. The course’s recordings cover only the course’s documents. So for your documents, you either call a model with your own key (--live, at your own cost), or you use no model. Both are acceptable. Say which one you chose.
• Python 3.12 and the course’s requirements.txt. Nothing else to install.
• You can do the assignment in more than one session. Run the tests after each change.

The situation: a new kind of document

Larkfield’s help desk gets a new kind of document. Grace asks the course’s question again: “Can the system read it, show us exactly where each value came from, and send us only the ones it is not sure about?” Choose one of the two options.

Option A: receipts from a new shop. Larkfield adds a second partner shop. Its receipts have their own layout and their own words, for example NET, GST/HST and PAID BY. Customers send them as digital PDFs, scans and phone photos, like Larkfield’s receipts. Use the course’s receipt contract, or a new version of it if the new shop needs one.

Option B: fields from policy documents. The purchasing team wants a few facts from each supplier’s warranty policy. They are the supplier, the policy version, the date it takes effect, the warranty period in months, and who pays for return shipping. The policies arrive as digital PDFs with two columns, as scanned pages, and sometimes as a page turned on its side. Write a new contract for them, for example policy-v1.

Every document in your set is made by your own script, as the course’s documents are. No real customer, shop or supplier may be in it. Say so in your report.

Part 1: your documents and your contract

Uses Set up the extraction project, Define the extraction contract and Document types.

1. Write the contract first, in docs/extraction-contract.md and in contract.py. For each field, give its format, whether it is required, and the reasons that it may be absent (not_printed, cut_off, unreadable). Say how each field is normalised. Give the contract a new version name if you change the course’s one.
2. Write a seeded script, make_set.py, that makes your documents and their truth in a separate folder. You may reuse the course’s generator: generate.make_receipt, generate.receipt_pdf, generate.scan, generate.photo, generate.image_pdf, generate.jpeg and generate.policy_pdf. For Option A, add a function that draws the new shop’s layout.
3. Make at least 20 documents of at least three document types. For Option A, use digital PDFs, scans and photos. For Option B, use digital PDFs, scanned pages and turned pages. Include at least two documents that do not print an optional field. Include one in French if your option allows it.
4. Add at least four problem files: a damaged file, an exact copy, a file whose name does not match its content, and a picture that is cut off or cannot be read.
5. Write the truth of every document in truth.jsonl, one line per document, with every field that the document prints. A field that the document does not print gets no value. Run the script twice, and check that it writes the same bytes, as generate --check does.
6. Read every document once yourself, and check its truth by eye. Write in your report who labelled the documents and how.

Part 2: format-aware ingestion

Uses Document types, Input handling and Image preparation.

1. Run your documents through intake. Decide each format by the first bytes, never by the name. Stop the problem files with a reason that a person understands.
2. Route each page: text layer, OCR, or stop. A file with a digital page and a scanned page is routed page by page.
3. Prepare the pictures before OCR: crop, turn and straighten. Keep the original file, and map every box back onto it.
4. Print a table of the routes and the intake decisions for your whole set, and put it in your report.

Part 3: extraction with evidence

Uses Text and OCR paths, Layout and meaning, Sections and hierarchy (Option B), Provenance representation, Input and prompt design, Combine methods and Handle visual uncertainty.

1. Write rules for your new layout or your policy fields. Run them on the text layer and on the OCR lines. Keep the rules small: a list of the words that your documents print.
2. Give every found value its evidence: the file, the page, the box and its unit, and the text seen there. The unit is points on a PDF page and pixels on a picture file. For Option B, add the section path of the value, such as “Warranty > Period”.
3. Keep every value that is not available explicit: no value, the status absent, and a reason that your contract allows. Never compute a value that the page does not show.
4. If you use a model: send the right page, and crop when the document is small on the picture. Use a strict schema with rules for absent values. Call the model only when the rules are not enough. Ground each of its values on a line of the page, or mark it unverified. Store every answer with the model name, the tokens, the seconds and the cost. Choose a spending limit before you start, stay under it, and report what you spent.
5. If you use no model: the documents that your rules cannot read stay incomplete, and they go to review. Say how many, and what a model would add.
6. Compute the uncertainty signals of each page: too few characters, a low mean score, small text. Flag any absent value whose reason your contract does not allow.

Part 4: validation, review and corrections

Uses Business validation, Confidence and review routing and Correction workflow.

1. Write the validation checks for your contract, each with a severity. An error sends the document to a person, and a warning goes with it. Write the reason for each check in a table. Add at least one check that compares fields with each other. Examples are a total that adds up, or a start date that comes before an end date.
2. Find duplicates by content, not by the picture.
3. Route to review. Every document with a reason (an error, a conflict or an unverified value) goes to a person. The others go when their certainty is under your threshold. Compute the certainty from things that you measure, not from a model’s own confidence. If you use a model, measure whether its confidence is calibrated on your set.
4. Run a threshold sweep on your set, and choose the threshold from it. Say what each step of the threshold adds to the queue, and whether it catches an error.
5. Write docs/review-queue.md for your documents: what goes to the queue, what the reviewer sees, and what the reviewer can do.
6. Record at least two corrections with the course’s corrections module, or with your own code under the same rules: append-only rows, a reviewer, a note and a version. Then validate the current values again.

Part 5: error analysis across document types

Uses Field-level metrics, Operational constraints and Integration and maintenance.

1. Score every field of every document with the five outcomes: correct, missing, wrong, correct absent and invented. Count exact and normalised matches apart.
2. Build the evaluation matrix: fields down, document types across. Count the complete documents for each type: every required field correct, and nothing invented.
3. Compare at least two systems on the same documents. Examples are OCR raw against OCR prepared, the rules alone against the rules with the model, or two prompts. Change one thing at a time.
4. For every wrong, missing or invented value, write one line: the document, the field, the cause and the fix that you would try. The cause is the picture, the OCR, the rules, the model or the contract. Group the lines by cause.
5. Measure the time per page of each path on your computer. If you used a model, give its tokens and cost per page. Budget a day of the volume that you assume, with each number labelled measured, recorded or assumption.
6. Make a regression set with at least one document of each type and of each failure that you saw. Change one thing in your pipeline, and give it a new version. Run the set, and explain each difference, or show that there is none.

No claim beyond your documents. Your set is small and made by your own script. Your report must not say how well the pipeline reads real customer documents. Claims about your set are fine, with their conditions: “on my 24 documents, preparation made the two turned scans complete”. A claim about real documents is not. If someone needs one, your report says which labelled real documents must measure it.

The failure paths you must check

A pipeline that works on clean documents is not enough. Show each of these with a test or a recorded run, and say which. Label each constructed case as constructed.

1. The damaged file and the exact copy stop at intake with a reason, and nothing is extracted from them.
2. The file whose name does not match its content is read by its real format.
3. A turned or crooked picture is prepared, and its boxes are drawn on the original file in the right places.
4. A field that the document does not print is stored as absent with the reason not_printed, and the evaluation counts it as correct absent.
5. A cut-off or unreadable picture gives absent values with the reason cut_off or unreadable. No value is computed or guessed.
6. A value that breaks a validation rule sends the document to review with the reason in words. For example, make one receipt whose printed total does not add up (constructed).
7. A correction adds a row and keeps the extracted value. An UPDATE of an extracted row is refused.
8. After a change to the code and a new version, pipeline status finds the old results, and the regression set runs.

Fresh-environment run

Before you submit, prove that the project works from nothing, on your computer:

1. Copy the project to a new folder, without .venv, .env, out/ and your generated documents.
2. Make a new virtual environment, and install from requirements-dev.txt.
3. Run make_set.py, and check that it writes the same files as before.
4. Run every test.
5. Run the whole pipeline on your set, then the evaluation. Compare the numbers with your report.

Paste the commands and their outputs into your report. If a step needed something that is not in the README, fix the README, and run the steps again.

What to submit

Your project folder (without .env, .venv, out/ and the generated documents), with make_set.py, your contract, code and tests, docs/review-queue.md, docs/evaluation-report.md and REPORT.md:

# Larkfield: <your document type> with evidence and review
## The documents and the contract   (the option, the set and its types, the problem files, the contract
                                     with its absent reasons, who labelled the truth and how)
## Ingestion                        (the routes and intake table, how the format is decided,
                                     the preparation steps, the boxes on the original)
## Extraction and evidence          (the rules, the model or no model and why, the evidence of a value,
                                     grounding, absent values with reasons, the uncertainty signals)
## Validation and review            (the checks with severities and reasons, duplicates, the certainty,
                                     the threshold sweep and the choice, the queue design, corrections)
## Error analysis                   (the matrix, complete documents by type, the two systems compared,
                                     every error with its cause and fix, time, cost and the day's budget,
                                     the regression set and one change)
## Failure paths                    (each of the eight, with the test or the run that shows it)
## Fresh-environment run            (the commands and their outputs)
## Limits                           (synthetic documents, your computer, what you reused, what you did not test)

How it is marked

A reviewer marks each required item demonstrated, partially demonstrated or not yet demonstrated, with a concrete reason. To complete the course, all four must be demonstrated. You can improve an item and submit it again.

Required item	Demonstrated	Partially demonstrated, for example	Not yet demonstrated, for example
Format-aware ingestion	Each format is decided by its bytes. Every problem file stops with a clear reason. Pages are routed one by one. Pictures are prepared, and boxes map back to the original. The routes table covers the whole set.	Intake works, but one problem file passes. Or the boxes are drawn on the prepared picture, not on the original.	The format is decided by the file name. Or a damaged file crashes the pipeline.
Field-level validation and source references	Every found value has its file, page, box and unit (or sheet and cell). Absent values have a reason that the contract allows. The checks have severities and reasons, and at least one compares fields. Model values are grounded or marked unverified.	The evidence is there, but without the unit, or only for some fields. Or every problem is an error, so the queue fills with right documents.	A value without evidence is shown as found. Or a missing value is filled with a computed or guessed one.
Review queue design	Reasons always route. The certainty is measured, and the threshold comes from a sweep on your set. The design says what the reviewer sees and can do. Corrections are append-only, with a reviewer, a note and a version, and the current values are checked again.	The queue works, but no measurement supports the threshold. Or corrections have no note.	Documents are routed by the model’s own confidence, with no check of its calibration. Or a correction overwrites the extracted value.
Error analysis across document types	Every field gets one of the five outcomes, invented values included. The matrix is by type and field, with complete documents. Two systems are compared on the same documents. Every error has a cause and a fix. The report says what the measurement cannot show.	The matrix is there, but invented values are not counted. Or the errors are counted, but not explained.	One accuracy number for the whole set. Or the truth was copied from the pipeline’s own output.

Whatever else is strong, the work is not complete in two cases: the pipeline shows an invented or computed value as found, or a correction erases an extracted value. A high accuracy does not make up for a plausible value with no evidence, or for a history that nobody can check.

The course’s own results can help you judge yours, but they are not targets. In Combine methods, the combined pipeline read 427 of 435 printed values right, with nothing wrong or invented. It called the model for 9 of 54 receipts. In Input and prompt design, the plain prompt invented 11 order numbers, and the precise prompt none. In Confidence and review routing, 11 of 54 receipts went to review at the threshold 0.90, and no wrong total reached the agents. Your documents are different.

AI assistants

You may use an AI assistant to practise, to explain an error, or to suggest a rule, a check or a test. Check each suggestion, and be ready to explain your final work. For each rule, check and threshold, say what it does, why you chose it, and which run proves it. Mark in your report the decisions that are your own, and the suggestions that you rejected, with the reason. An assistant may not write your truth labels, and it may not produce your evidence. Every count, score and cost in your report comes from a run on your computer.

Common mistake

A perfect score from labels that the pipeline wrote

Symptom: the evaluation shows 100% on every field and every document type. Then a reviewer opens three photos and finds a wrong date and a misread amount.

Cause: the truth was made from the pipeline’s own output: run the pipeline, save the values, and call them the labels. The evaluation then compares the pipeline with itself, so every error scores as right. This is a constructed case.

Fix: write the truth from your generator, which knows each value because it printed it, and check it by reading the documents. Never change a label to match the pipeline. If a label is wrong, correct it with a note, as you correct an extraction, and run the evaluation again.

Recap

In this course, you built a pipeline that reads documents and can show where every value came from:

• Look inside, then route: decide the format by the bytes, stop bad files at the door, and send each page to the text layer or OCR.
• One record for every path: lines with boxes and scores, prepared pictures mapped back to the original, and a contract with its absent reasons.
• Structure and evidence: sections, tables and spreadsheet cells, and each value with its file, page and box.
• The model on purpose: a precise prompt, the right input, the rules first, and every model value grounded on the page or marked unverified.
• Checks and people: validation with errors and warnings, routing by reasons and a measured certainty, and corrections that never erase.
• Measure and maintain: five outcomes per field, the matrix by type, a budget with its assumptions, versions, deletion with a log, and a regression set.

If you have not done one yet, choose a case study for the certificate: Read real receipts or Extract a table from a public report.