Input handling

What you will be able to do

Check every file before anything reads it. A damaged, locked, oversized or repeated file then stops at the door with a reason that a person can read. It does not crash the reader or reach an agent twice.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• How the project finds a file’s real format from its first bytes (route.sniff), from The idea: look inside, then route in Document types.
• The 65 documents of the course and their IDs (D001 to D065), from Larkfield’s document set in Set up the extraction project.
• The fields Larkfield needs from a receipt, from Larkfield’s contract in Define the extraction contract.

What you need on your computer

• Your project ~/projects/doc-extract at the end of Module 1: with the virtual environment active, python -m pytest -q prints 40 passed. If your project is broken, copy the snapshot end-of-m01 as the snapshots README says.
• The course’s documents in data/documents/: python -m doc_extract.generate --check prints All 65 files match the course's checksums.
• The labs repository in ~/projects/nextia-learning-labs, from the setup lesson. Get its newest files with git -C ~/projects/nextia-learning-labs pull.

Every document in this course is made by the course’s generator. No real customer, shop receipt or card number is in it. Tested with Python 3.12.13, pypdf 6.20.0, pypdfium2 5.14.0 and Pillow 12.3.0 on macOS (Apple silicon). The Windows and Linux commands are not tested.

Get the files of Module 2

The three lessons of this module use files that work together: the checks of this lesson, three ways to read a page, and image preparation. So you copy them once, now, from the snapshot end-of-m02. Each lesson then opens its own file.

1. Go to the projects folder:

   cd ~/projects

2. Copy the snapshot’s files over your project. Your data/, out/ and .venv folders stay. The copy replaces README.md and .env.example with their Module 2 versions.

   macOS / Linux

   cp -R nextia-learning-labs/C13/snapshots/end-of-m02/. doc-extract/

   Windows (PowerShell)

   Copy-Item -Recurse -Force nextia-learning-labs\C13\snapshots\end-of-m02\* doc-extract\

3. Go into the project and activate the virtual environment (source .venv/bin/activate on macOS and Linux, .venv\Scripts\Activate.ps1 on Windows).

4. Run the tests. They make their own copy of the documents and run OCR on a few pages, so they take about half a minute:

   python -m pytest -q

   The last line is 63 passed. You install nothing new: requirements.txt already had every package of the course.

A mail room at the door

Think of the mail room of a large office. Before any letter goes up to a desk, a clerk looks at each envelope. A parcel that is too big goes to another counter. An empty envelope goes back. A torn envelope that lost its content cannot be read. A locked box needs a key that the clerk does not have. And if the same letter arrives twice, the second copy does not go upstairs, or two people answer it.

The clerk does not read the letters. The checks are fast, and they happen once, at the door. Because of them, nobody upstairs loses time on a parcel they cannot open. And when the clerk sends something back, the note says why: “too large”, “damaged”, “locked”.

The problem: the first reader that meets a bad file

Larkfield’s tickets bring files that customers attach, and customers attach everything. The course’s set has seven problem files on purpose, D059 to D065, of the kinds that the help desk sees. Without a check at the door, the first program that reads a file meets the problem, and it does not explain it well.

Two real examples, from the course’s files. D064 is called receipt-scan.pdf, but it is a photo in JPEG format with the wrong name. Give it to PDFium, the library that draws PDF pages as pictures, and the program stops:

pypdfium2._helpers.misc.PdfiumError: Failed to load document (PDFium: Data format error).

D061 is a PDF whose download stopped halfway. pypdf, the library that reads the structure of a PDF, stops with:

pypdf.errors.PdfReadError: EOF marker not found

Neither message tells Grace’s agents what to do. Worse, an error in the middle of a batch can stop the files after it too. And some problems give no error at all. D059 is an exact copy of D001: the customer attached the same receipt twice. Both copies read perfectly, so both reach an agent, and the same refund can be checked twice.

The first example was a real bug. During the course’s development, a JPEG named .pdf crashed the project’s page renderer. The fix is the rule of this lesson: decide the format by the content, never by the name. files.is_pdf now reads the first bytes of the file everywhere.

The idea: input handling

Input handling (or intake) is the set of checks that every file passes before any reader opens it for real. Like the clerk, it does not read the content. It asks a few short questions, in a fixed order. The first problem stops the file with a status, a short word that code can count, and a reason, a sentence that a person can read. The order goes from the cheapest check to the most expensive one, so that a bad file costs as little as possible.

doc_extract/intake.py asks these questions:

1. Size. Is the file bigger than the limit, 8 MB by default (MAX_FILE_MB)? A huge file slows every later step, and a model call would cost more for it. Status too_large.
2. Empty. Does the file have 0 bytes? Status empty.
3. Format. Do the first bytes say PDF, JPEG, PNG, TIFF or XLSX? Every format starts with its own short signature, its magic number: %PDF- for a PDF, the bytes ff d8 ff for a JPEG. The name of the file is not used. Status unsupported.
4. Opens. Can the right library open it? pypdf opens a PDF, and PDFium draws its first page very small; Pillow decodes a picture; the spreadsheet’s zip is tested. Status corrupt, or password for a PDF that needs a password.
5. Exact duplicate. Are the bytes the same as a file that arrived before? Status duplicate, with the name of the first copy.
6. Name. Does the extension match the content? If not, the status is wrong_extension, but the file is accepted and read as what it really is.

The last check is the only one that does not stop the file. A wrong name is not a reason to refuse a good receipt. It is a reason to trust the content and to write down what was wrong.

Exact copies: SHA-256

How does the program know that two files have the same bytes, without keeping every file it ever saw? It computes a hash: a short summary of the bytes, always of the same length. The project uses SHA-256, which turns any file into 64 hexadecimal characters. The same bytes always give the same 64 characters. Change one byte, and about half of the characters change. So the program keeps one hash per file, and a new file whose hash is already known is a copy.

Here are the real hashes of three of the course’s files:

758bcf84593dc3430f0fddeaf02073530b1a3996ea3eab0458cb5c560456740e  receipt-lk-986875.pdf
758bcf84593dc3430f0fddeaf02073530b1a3996ea3eab0458cb5c560456740e  receipt-lk-986875 (1).pdf
bcee48048deb959a629771227ac8e465673456e69a6252051d16d7f71ae8db46  scan0002.jpg

The first two are D001 and D059: the same bytes, the same hash. The name (1) is what a computer adds when you save a file twice. The third file is different, so its hash has nothing in common with the others.

A tiny example: five files at the door

Before you run the checks on all 65 files, follow five of them by hand. For each file, the table shows its size, its first bytes and the decision.

File	Size	First bytes	First problem	Status
receipt-lk-986875.pdf (D001)	2,366 bytes	%PDF-	none	accepted
receipt-lk-986875 (1).pdf (D059)	2,366 bytes	%PDF-	the hash 758bcf84… is already known	duplicate
receipt-scan.pdf (D064)	289,721 bytes	ff d8 ff (JPEG)	the name says PDF, the content says JPEG	wrong_extension, accepted
receipt-download.pdf (D061)	1,158 bytes	%PDF-	it does not open: a PDF ends with %%EOF, and this one stops before	corrupt
notes.txt (D065)	41 bytes	Customer says the…	not a PDF, picture or spreadsheet	unsupported

Two details matter. D061 starts like a good PDF: its first bytes are right, so only the fourth check, “does it open?”, finds the problem. And the order of arrival matters for duplicates. D001 arrived first, so D001 is accepted and D059 is the duplicate. If D059 had arrived first, D001 would be the copy. One of the project’s tests checks exactly this.

The intake checks, in their order. The first problem stops the file; only a wrong name lets it through. The counts are from the course's 65 files.

Text description of the diagram

A file arrives; the course has 65. The first question is the size: a file bigger than 8 MB stops as too_large. In the course’s set, that is D063, a 9.1 MB uncompressed TIFF. The second question: does the file have 0 bytes? None does here. The third: do the first bytes say PDF, JPEG, PNG, TIFF or XLSX? D065, a text file, stops as unsupported. The fourth: does the file open? D061 stops as corrupt and D062 as password. The fifth: does the file have the same SHA-256 hash as a file that came before? D059, a copy of D001, stops as duplicate. The last question: does the name match the content? For 59 files it does: accepted. For D064, a JPEG named .pdf, it does not: wrong_extension, but the file is accepted and read as a JPEG. So 60 files go on to be read.

Run the checks on Larkfield’s files

Now run the checks on the whole set, in the order the files arrived. intake prints only the files that are not plainly accepted, then the counts.

1. Run the intake checks:

   python -m doc_extract.intake

   D059  receipt-lk-986875 (1).pdf          duplicate        the same bytes as receipt-lk-986875.pdf
   D061  receipt-download.pdf               corrupt          EOF marker not found
   D062  receipt-protected.pdf              password         the PDF needs a password to open
   D063  scan-raw.tif                       too_large        9.1 MB is over the limit of 8.0 MB
   D064  receipt-scan.pdf                   wrong_extension  the name ends in .pdf, but the content is jpeg
   D065  notes.txt                          unsupported      the content is not a PDF, image or spreadsheet (unknown)
     accepted 59, corrupt 1, duplicate 1, password 1, too_large 1, unsupported 1, wrong_extension 1

   Five files stop and one goes on with a note. The 65 files took about 0.3 seconds in all.

2. Open doc_extract/intake.py and find check(). Read it from top to bottom. Each return is one status of the list above, in the same order.

3. Try a much smaller limit, 0.2 MB, to see the size check at work:

   python -m doc_extract.intake --max-mb 0.2

   The last line shows that 17 files are now too large:

     accepted 44, corrupt 1, duplicate 1, password 1, too_large 17, unsupported 1

   The 17 are 14 of the scans, the warranty claim (its page 2 is a scan), D063 and D064. Look at one of the lines above the counts: D020 scan0002.jpg too_large 0.2 MB is over the limit of 0.2 MB. The file has 224,481 bytes, which the message rounds to 0.2 MB. The check compares bytes, so it is right; only the message is short. Also notice that D064 now stops as too_large, before its name is checked. The first problem wins.

Sam asks: “Why does a corrupt PDF stop, but a wrong name does not?” Mei answers: “Nobody can read a corrupt file. We send it back with the reason, and the agent asks the customer for the file again. A file with a wrong name can be read perfectly, if we trust its bytes. Refusing it would only annoy the customer.”

Amira asks the next question: what does Larkfield do with a stopped file? In this course, the status and the reason go back with the ticket, and the file does not enter the pipeline. In Integration and maintenance, the whole pipeline keeps these decisions with the file’s other records.

Near duplicates: the same receipt, scanned twice

Exact copies are easy. But D060, scan-again.jpg, is a second scan of the same paper receipt as D020. The customer scanned it again because the first scan looked crooked. The two pictures show the same receipt, but almost every pixel is a little different. So the bytes are different, the hashes are different, and intake accepts both:

bcee48048deb959a629771227ac8e465673456e69a6252051d16d7f71ae8db46  scan0002.jpg
18942a38df8196d71e47929c4fae68c753c8dcd4aefc1ff3c856d6926ad3cdfd  scan-again.jpg

This is a near duplicate: two files with the same content but different bytes. Can a program see that two pictures look alike?

The idea: a fingerprint of the picture

Think of how you would compare two photos of the same room from far away. You would not compare every detail. You would look at the big shapes: dark here, bright there. An image fingerprint does the same. The project’s fingerprint, intake.fingerprint, is a difference hash. It crops the page to its content and shrinks it to a tiny grey picture of 17 × 16 pixels. Then it compares each pixel with its right neighbour: a brighter pixel gives the bit 1, a darker or equal one gives 0. Sixteen rows of 16 comparisons give 256 bits. Two fingerprints are compared by counting the bits that differ: 0 means “looks the same”.

A tiny example with one row of five grey pixels (0 is black, 255 is white). These values are made up to show the rule:

	Pixel 1	2	3	4	5	Bits (each pixel against the next)
First scan	200	180	180	90	240	1 0 1 0
Second scan, a little darker	190	172	171	85	230	1 1 1 0

The second scan is darker everywhere, but most comparisons stay the same, because they compare neighbours, not absolute values. Only pixels 2 and 3, which were equal, now differ by 1, and their bit flips. One bit of four differs. That is why a fingerprint survives a darker scan or a smaller copy.

What the fingerprints show on Larkfield’s files

Now the real result. intake --fingerprints prints the 12 closest pairs of pages in the set:

python -m doc_extract.intake --fingerprints

  0 of 256 bits differ: receipt-lk-986875.pdf / receipt-lk-986875 (1).pdf
  1 of 256 bits differ: receipt-lk-139958.pdf / order-us-lk-159455.pdf
  1 of 256 bits differ: receipt-lk-848917.pdf / receipt-lk-139958.pdf
  2 of 256 bits differ: order-us-lk-159455.pdf / order-us-lk-517444.pdf
  2 of 256 bits differ: receipt-lk-237469.pdf / receipt-lk-139958.pdf
  2 of 256 bits differ: receipt-lk-848917.pdf / order-us-lk-159455.pdf
  2 of 256 bits differ: receipt-lk-848917.pdf / order-us-lk-517444.pdf
  3 of 256 bits differ: receipt-lk-139958.pdf / order-us-lk-517444.pdf
  3 of 256 bits differ: receipt-lk-237469.pdf / order-us-lk-159455.pdf
  3 of 256 bits differ: receipt-lk-237469.pdf / order-us-lk-517444.pdf
  3 of 256 bits differ: receipt-lk-237469.pdf / receipt-lk-848917.pdf
  6 of 256 bits differ: receipt-lk-246283.pdf / receipt-lk-946210.pdf

The first pair is the exact copy: 0 bits. But every other pair is two different receipts. receipt-lk-139958.pdf (D016) and order-us-lk-159455.pdf (D052) differ by 1 bit of 256. They are two online orders with different customers, items and totals, even in different currencies. And the real second scan, D060 against D020, is not in the list at all. To see its distance, start Python in the project folder with python, and type:

from doc_extract import documents, intake
a, b = documents.get("D020"), documents.get("D060")
print(intake.distance(intake.fingerprint(a.path, "jpeg"), intake.fingerprint(b.path, "jpeg")))

10

The second scan of the same receipt differs by 10 bits. Different receipts are as close as 1 bit. In the whole set, 18 pairs of different receipts are closer together than the real near duplicate.

What the fingerprint sees. Top: two different receipts on the same template, 1 bit apart. Bottom: one receipt scanned twice, 10 bits apart. The grey copies hide every value; only the template is left.

Text description of the diagram

The picture has two rows. Each row shows two pages, their tiny 17 × 16 grey copies, and a grid of 256 squares between them, one per bit. A red square is a bit that differs. In the top row, D016 and D052 are two different online orders from Larkfield, with the same green header, the same columns and different values. Their grey copies look almost identical, and only 1 square of 256 is red. In the bottom row, D020 and D060 are the same order confirmation, scanned twice. Their grey copies also look alike, and 10 squares are red. At 17 × 16 pixels, the numbers, names and dates disappear. What is left is the template: the header bar, the columns, the white space. Receipts made from the same template look the same.

The picture shows why. At 17 × 16 pixels, no number, name or total is left. Only the template is left: the header bar, the table, the white space at the bottom. Larkfield prints every online order with the same template, so different receipts look the same. A second scan, slightly crooked and darker, looks less like its first scan than two different receipts do.

So a threshold cannot work. Set it at 10 bits to catch D060, and 18 pairs of different receipts are flagged as duplicates. An agent would then reject real refunds. Set it lower, and D060 passes anyway.

The decision: compare the content, after extraction

So the course splits the job. Intake stops only exact copies, by their SHA-256 hash, because that test is cheap and never wrong. Near duplicates are found later, by their content, once the values are read: the same order number, or the same shop, date and total. That check lives in Business validation. There, the whole pipeline flags D060 as a possible duplicate of D020, and a person decides. The fingerprint stays in the project only to show this result: --fingerprints is a tool for looking, not a rule.

When to use these checks, and what they cost

Check every file at the door, always. The checks are cheap: about 0.3 seconds for all 65 files on the course’s computer. Only the opening check does real work, and it draws the first page at a tenth of its size. The fingerprints took about 0.5 seconds.

Choose the size limit from what the next steps can afford, not from a guess. The course’s limit, 8 MB, lets every normal scan and photo through: the biggest accepted file is 0.6 MB. Only D063, an uncompressed TIFF of 9.1 MB, stops. A limit of 0.2 MB would refuse 15 good receipts and the warranty claim, so agents would ask customers for files they already sent. If your files differ, change the limit in .env with MAX_FILE_MB.

These checks cannot do a few things. They cannot tell whether a picture is readable: D046, a blurred photo, opens without an error and passes. Reading it is the job of the next lessons, and Handle visual uncertainty treats unreadable pages. They cannot see near duplicates, as you saw. And here, the registry of hashes lives in memory for one run. A real system keeps the hashes in its database, so that it still finds a copy that arrives next week.

Common mistake

Trusting the file name

Symptom: A batch of attachments stops halfway with PdfiumError: Failed to load document (PDFium: Data format error). The file is called receipt-scan.pdf, so the code sent it to the PDF renderer.

Cause: The name was wrong. The file is a JPEG: its first bytes are ff d8 ff, not %PDF-. Phones, email programs and customers rename files, and a name is only a label that anybody can change. Code that decides by the extension sends the file to the wrong reader.

Diagnose: Look at the first bytes. python -m doc_extract.route D064 prints the real format and says what the name claims:

D064  receipt-scan.pdf: format jpeg  <- the name says .pdf, 1 page(s), kind scan
  page 1: route ocr   (a scanned image (white page); pictures cover 100%)

Fix: Decide the format by the content everywhere, not only at the door. In the project, route.sniff reads the magic number, and files.is_pdf reads the first five bytes before it draws a page. A wrong name becomes a note (wrong_extension), and the file is read as what it is.

Consequence in AI work: a pipeline that trusts names fails on real files. It is hard to debug, because the error comes from a library deep inside, far from the cause. Worse, a reader that does not crash can read garbage and pass it on as text.

Your turn: the order of arrival

Change one condition: the order in which two files arrive. Suppose the customer’s second message, with receipt-lk-986875 (1).pdf (D059), reached Larkfield before the first one (D001). Predict what intake decides for each file. Then check it in Python (start python in the project folder):

from doc_extract import documents, intake
registry = intake.Registry()
for doc_id in ("D059", "D001"):
    r = intake.check(documents.get(doc_id).path, registry)
    print(doc_id, r.status, r.reason)

Show the answer

D059 accepted
D001 duplicate the same bytes as receipt-lk-986875 (1).pdf

The first copy that arrives is accepted, whatever its name, and the second one is the duplicate. The bytes are the same, so the program cannot know which one is “the original”. It does not need to: both are the same receipt. This is why the registry remembers the name of the first copy (same_as). An agent who sees “duplicate” can find the file that was kept. The project’s test test_the_first_copy_wins_the_order_matters checks this case.

Knowledge check

Check your understanding

Put the intake checks in order and say why the order goes from cheap to expensive. Which check is the only one that lets a file through, and why?

Show the answer

The order is: size, empty, format by the first bytes, does it open (corrupt or password), exact duplicate by SHA-256, and the name. The first checks read only the file’s size or its first bytes, so a bad file stops for almost no cost. Opening the file is the most expensive check, so it runs only on files that passed the others. The name check is the only one that lets the file through. A file with wrong_extension has good content, so the project reads it as what it really is. A wrong name is a note, not a reason to refuse a receipt.

Check your understanding

Predict and explain: Tomás proposes to stop near duplicates at intake with the image fingerprint, at 10 bits or fewer. What happens on Larkfield's files?

Show the answer

D060, the second scan of D020, is 10 bits away, so it would stop. But 18 pairs of different receipts are closer than that; D016 and D052 are only 1 bit apart. At 17 × 16 grey pixels, the values disappear and only the template is left, and Larkfield prints its receipts from a few templates. So the rule would refuse real receipts as “duplicates”. The course compares the content instead, after extraction: the same order number, or the same shop, date and total.

Check your understanding

Practical check: run the intake checks with a size limit of 0.6 MB. Which files are too large, and which of them would you not want to refuse?

Show the answer

Run python -m doc_extract.intake --max-mb 0.6. Two files are too large: D063 (scan-raw.tif, 9.1 MB) and D019 (scan0001.pdf, 639,633 bytes). The message rounds D019’s size: “0.6 MB is over the limit of 0.6 MB”. D063 is the uncompressed TIFF that the course stops on purpose. D019 is a normal scanned receipt: refusing it would send the agent back to the customer for no reason. A limit must come from what the next steps can afford, and it must let normal files through. The default, 8 MB, does.

Recap

• Input handling checks every file before any reader opens it, from the cheapest check to the most expensive. The checks are size, empty, format by the first bytes, opens (corrupt, password), exact duplicate and name. The first problem stops the file with a status and a reason that a person can read.
• On Larkfield’s 65 files, 5 stop (D059 duplicate, D061 corrupt, D062 password, D063 too large, D065 unsupported). D064, a JPEG named .pdf, is accepted and read as a JPEG. 60 files go on.
• Decide the format by the content, never by the name. A wrong name crashed the project’s renderer once.
• SHA-256 finds exact copies for almost no cost. Near duplicates have different bytes. An image fingerprint puts D060 10 bits from D020, but 18 pairs of different receipts are closer. So the course finds near duplicates by their content, after extraction.

Next, you read the text out of the accepted files in three ways, and compare them on the same pages.

Next: Text and OCR paths