Image preparation

What you will be able to do

Make a picture easy to read before OCR: turn it upright, crop it to the document, straighten it and enlarge small text. Measure what each fix changes, and keep the original file, so that every value can still be shown on the picture that the customer sent.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• OCR, its lines with boxes and scores, from The idea: three paths, one record, and compare, from Compare the three paths on one page, both in Text and OCR paths.
• Why a box with its file and page is the evidence of a value, from The idea: an extraction contract in Define the extraction contract.

What you need on your computer

• The project ~/projects/doc-extract with the files of Module 2 and the virtual environment active: python -m pytest -q prints 63 passed.
• A program that opens PNG pictures, to look at the prepared pictures in out/prepared/. VS Code opens them.

Tested with Python 3.12.13, Pillow 12.3.0, NumPy 2.5.4 and RapidOCR 3.10.0 on macOS (Apple silicon). The commands are the same on Windows and Linux, but they are not tested there.

A newspaper handed to you upside down

Somebody hands you a newspaper upside down. You do not try to read it that way: you turn it first. A small note lies on a big desk in a photo: you look only at the note. The print is tiny: you hold it closer, or you use a magnifier. You do these things without thinking, and you do them before you read.

You also do not cut the newspaper. If somebody later asks “where did you read that?”, you point at the place in the newspaper that you got. You do not point at your own turned and enlarged copy.

The problem: OCR on a page that lies on its side

In the last lesson, OCR got 1 of 9 fields right on D025, a scanned PDF. Look at what it read:

python -m doc_extract.ocr D025

scan0007.pdf, page 1: 28 lines by rapidocr 3.10.0 in 0.93 s (27 with a score under 0.80)
  0.77 [  1385    530   1416    615]  10:35
  0.76 [  1593    531   1624    614]  34.50
  0.79 [  1659    526   1700    615]  34.50
  0.74 [  1691    524   1734    605]  1..73
  0.75 [  1731    526   1767    605]  2.442
  0.79 [  1765    528   1801    675]  CA 3.65
  0.58 [  1810    543   1827    636]  一-----
  0.80 [  1837    530   1868    615]  38.65
  0.62 [  1903    602   1944   1015]  rhcce r Reek c xept
  0.59 [  1935    622   1980    997]  o  sn xnty.
  0.68 [  1315    663   1349    958]  Vaar   1
  0.75 [  1201    701   1243    920]  IARILD
  0.72 [  1241    708   1280    910]  Homr  eaden
  ...
  0.62 [  1382    835   1421   1093]  Dat00: 20-13
  0.77 [  1416    836   1456   1093]  Oord 359
  ...
  0.65 [  1664    960   1698   1092]  Subttal
  0.73 [  1767   1004   1798   1090]  TOTAL

IARILD for LARKFIELD, Dat00: 20-13 for the date, CA 3.65 for CAD 38.65, and a Chinese character for a dashed line. 27 of the 28 lines score under 0.80. The boxes are tall and narrow: the receipt lies on its side. OCR expects text from left to right, so it reads the letters sideways and guesses.

D026 lies upside down, and OCR read its vendor as Docu n. That is not a missing value but a wrong one, which the rules would pass on.

The idea: image preparation

Image preparation changes a copy of the picture, before OCR, so that the text is upright, straight, large enough and alone on the picture. doc_extract/prepare.py has four steps:

1. Crop to the document. On a white scanner page, keep the printed part. On a photo, keep the bright paper and drop the darker table around it.
2. Turn the page upright. The program does not know which way is up, so it tests it. It reads a small copy with OCR. If the result reads badly, it tries the three other quarter turns (90, 180 and 270 degrees) and keeps the best one.
3. Straighten a page that is a little crooked. This step is called deskew (a “skew” is a slope). The slope is the median slope of the long OCR lines; the picture turns back by that angle when it is more than 1 degree.
4. Enlarge small text. OCR reads best when a line of text is about 32 pixels high. If the median line is under 24 pixels, the picture is enlarged to reach about 32, by at most 2.5 times.

Each step is written down: the crop box, the turn, the deskew angle, the scale. The original file is never changed. With the steps, the program can take a box found on the prepared picture and compute where it is on the original picture. So the evidence of every value still points at the file that the customer sent.

How “reads badly” is decided

On the small copy, the program multiplies the mean OCR score by the share of boxes that are wider than they are tall. Text on its side gives tall boxes and low scores. Under 0.85, it tries the other turns, and a turn wins only if it reads clearly better (by 15%). So a good page is read once, and only a bad one costs three more quick reads.

RapidOCR also has a classifier that turns single lines that look upside down. The project switches it off: during the course’s development, it read 99,00 on a photo tilted by 12 degrees (D040) as 00'66. Turning the whole page once, after a test, is safer.

A tiny example: a box goes back to the original

How does a box on the prepared picture go back to the original? Undo the steps in reverse order. Here is a made-up case with two steps, so you can follow every number.

The original picture is 1,000 × 800 pixels. Preparation crops it to the box from (100, 50) to (700, 650), then enlarges the crop 2 times. On the prepared picture, OCR finds the total in the box from (200, 300) to (260, 320).

Step	Left, top	Right, bottom
Box on the prepared picture	200, 300	260, 320
Undo the scale: divide by 2	100, 150	130, 160
Undo the crop: add the crop’s corner (100, 50)	200, 200	230, 210

So the total is at (200, 200) to (230, 210) on the original. The project’s function gives the same result:

from doc_extract import prepare
steps = {"original_size": [1000, 800], "crop": [100, 50, 700, 650], "scale": 2.0}
print(prepare.point_to_original(200, 300, steps), prepare.point_to_original(260, 320, steps))

(200.0, 200.0) (230.0, 210.0)

A turn and a deskew are undone in the same way, with a rotation instead of a division. point_to_original handles all four steps. A test sends a dot through a crop, a quarter turn, a 3-degree deskew and a ×2 scale, and it comes back within 3 pixels.

Image preparation works on a copy and writes down every step, so every box can go back onto the original picture.

Text description of the diagram

The program crops a copy of the original, tests it with a quick OCR and turns it if it reads badly. It straightens the copy if the lines slope more than 1 degree, and enlarges it if the text is under 24 pixels high. OCR reads the copy. The written steps are undone in reverse order, and every box goes back onto the original picture.

Prepare Larkfield’s turned scan

1. Prepare D025 and read the steps:

   python -m doc_extract.prepare D025

   scan0007.pdf, page 1: 2200 x 1700 -> 580 x 820: crop [1196, 520, 2016, 1100], turn 270
     prepared picture: out/prepared/D025-p1.png (the original file is not changed)

   The page was drawn at 200 dpi: 2,200 × 1,700 pixels, because the scanner page lies on its side. The crop keeps the receipt, 820 × 580 pixels, and the turn of 270 degrees makes it upright: 580 × 820.

2. Open out/prepared/D025-p1.png and look at it. It is an upright receipt, ready to read.

3. Read the page with preparation:

   python -m doc_extract.ocr D025 --prepare

   scan0007.pdf, page 1: 26 lines by rapidocr 3.10.0 in 2.86 s (0 with a score under 0.80)
     1.00 [  1207    703   1239    918]  LARKFIELD
     0.96 [  1249    715   1275    906]  Home & Garden
     1.00 [  1283    723   1310    897]  1020 Main St
     0.99 [  1319    666   1345    956]  Vancouver BC V6A 2W1
     0.98 [  1386    843   1415   1090]  Date: 2026-07-13
     1.00 [  1386    531   1414    612]  10:35
     0.99 [  1420    843   1449   1090]  Order: LK-536359
     ...
     1.00 [  1770   1008   1795   1089]  TOTAL
     1.00 [  1769    611   1795    671]  CAD
     1.00 [  1769    531   1796    614]  38.65

   Every line is now right, with scores of 0.90 or more. The boxes are tall and narrow again, at x from 1,200 to 2,000. They are on the original page, which lies on its side. OCR read the upright copy, and the project moved each box back. The page took 2.86 seconds instead of 0.93, because of the extra quick reads.

4. Compare the paths with preparation:

   python -m doc_extract.compare D025 --prepare

   The OCR column is now ok on every line, and the summary says ocr 9/9 correct, 2.86 s. Without preparation, it was 1 of 9.

5. Prepare D024, a scan made at a low resolution:

   python -m doc_extract.prepare D024

   scan0006.jpg, page 1: 637 x 825 -> 1359 x 1760: scale 2.133
     prepared picture: out/prepared/D024-p1.png (the original file is not changed)

   Nothing to crop or turn, but the text was small, about 15 pixels high, so the picture was enlarged 2.133 times.

D025 before and after preparation (left to right, or top to bottom on a phone): the original page and the crop, the prepared copy that OCR reads, and every box back on the original page.

Text description of the diagram

First: the original scanned page, wide and almost empty, with a small receipt on its side near the right edge. A blue rectangle marks the crop. Second: the prepared picture, the receipt upright and easy to read. Third: the original page again, with a green box around every line that OCR read on the prepared picture, moved back onto the sideways receipt.

What preparation changes, measured

One page is an example. The course read all 33 pictures of the receipt set (scans, photos and the claim’s scanned page) twice: raw and prepared. The OCR and the rules were the same. It counted the printed values right, by the condition of the picture.

• OCR on the raw picture
• OCR on the prepared picture

Printed values right before and after preparation, by condition: a page turned 90° or 180°, a small receipt on a scanner page, a plain scan, a photo. Turned pages go from almost nothing to everything; plain scans were already right; photos do not gain.

Text description of the chart

Printed values right, raw then prepared. Turned 90 degrees (D025, 8 values): 0% then 100%. Turned 180 degrees (D026, 9 values): 11.1% then 100%. A small receipt on a scanner page (11 scans, 85 values): 83.5% then 92.9%. Plain scans (5 pictures, 43 values): 100% both. Photos (15, 117 values): 88.0% then 87.2%.

Read it condition by condition:

• Turned pages gain most: 0 of 8 values to 8 of 8 on D025, 1 of 9 to 9 of 9 on D026.
• A small receipt on a big scanner page gains from the crop: 71 of 85 to 79 of 85.
• Plain, crooked (2 degrees or more), faded and low-resolution (100 dpi or less) scans were already all right: RapidOCR copes with them.
• Photos do not gain: 103 of 117 raw, 102 prepared. Their problems are blur (D046), a cut-off bottom (D045) or another shop’s words, not geometry.

Over all 33 pictures, preparation takes printed values right from 226 of 262 to 241 of 262. Over all 54 receipt pages, OCR goes from 392 of 435 to 408 of 435. That is why the course’s OCR path reads prepared pictures. The cost is time: the median page took 0.46 seconds raw and 0.96 seconds prepared. A turned page took about 2.3 to 2.4 seconds, because of the extra quick reads.

Preparation can also hurt

Look at the one photo value that preparation lost. On D045, OCR on the raw photo read the shop’s address as Toronto ON M5H 2M5. On the prepared photo, it read TorontoON M5H 2M5: one space less. The rules infer the currency CAD from a shop address in Canada, and they no longer recognise the address. So the currency goes missing:

python -m doc_extract.compare D045 --prepare

currency        CAD           (cannot read)   - WRONG         CAD ok

This is why you measure each change on the same pages, by condition, instead of trusting that “cleaner” means “better”. A change that helps turned scans a lot can cost a photo one value. The course keeps preparation, because it gains far more than it loses (392 to 408 of 435). And it records the loss.

Enlarging cannot add what is not there

Enlarging helps when the text is readable but small. It cannot add detail that the picture does not have. In Handle visual uncertainty, the course shrinks six receipts to a quarter of their size, 300 × 400 pixels. There, OCR on the raw pictures gets 29 of 51 values right, and on the prepared, enlarged pictures only 23. An enlarged blur is a bigger blur, and OCR then reads its shapes as wrong letters.

Keep the original

Why not save the prepared picture and forget the original? Because the original is what the customer sent, and what an agent, an auditor or the customer can open. When a reviewer corrects a value in Correction workflow, the correction points at the same file and box. A box on a private copy is a box that nobody else can check.

So the project’s rule is: evidence boxes are always on the original file. For a PDF page, the box is in points, or in pixels of the page drawn at a resolution that is stored with it (dpi). For an image file, it is in pixels of the image. The prepared picture in out/prepared/ is only for you to look at.

When to prepare, and what it costs

Prepare pictures before OCR: scans, photos and the scanned pages of mixed PDFs. A digital PDF’s text layer has no pixels to fix. Preparation fixes geometry: turn, crop, slope and size. It cannot fix blur, a missing part of the page or words that the rules do not know.

The costs are time and some complexity. A page takes about 0.5 seconds more, a page on its side about 2 seconds more, and you must record and undo the steps. Measure the effect on your own documents, by condition, and keep a record of each change: your scanners and phones are not the course’s.

Common mistake

Boxes drawn on the wrong picture

Symptom: The evidence box for the total of D025 shows an empty, white area in the top-left corner of the customer’s scan. The value itself, 38.65, is right.

Cause: The box came from the prepared picture and was stored as it was. On the prepared D025 picture, which is 580 × 820 pixels, OCR found TOTAL at (11, 574) to (92, 599). On the original page, which is 2,200 × 1,700 pixels and lies on its side, those numbers point at a blank corner. The box is right for a picture that nobody else has.

Diagnose: Compare the box with the size of the original page, and look at the page’s prepared steps. If they show a crop or a turn and the box was not moved, the box is on the wrong picture. Draw the box on the original to see it.

Fix: Move every box back with the recorded steps before you store it. In the project, ocr.read(..., prepare=True) calls prepare.to_original, which puts TOTAL at (1770, 1008) to (1795, 1089) on the original page: on the sideways receipt, where the word is.

Consequence in AI work: a wrong box is worse than no box, because it looks like evidence. A reviewer who checks it sees nothing, and stops trusting the pipeline.

Your turn: the upside-down scan

Change one condition: the direction. D026 is a French scan that lies upside down. Predict what prepare does with it, and how many fields OCR gets right without and with preparation. Then run:

python -m doc_extract.prepare D026
python -m doc_extract.compare D026
python -m doc_extract.compare D026 --prepare

Show the answer

scan0008.jpg, page 1: 1275 x 1650 -> 1275 x 1570: crop [0, 80, 1275, 1650], turn 180
  prepared picture: out/prepared/D026-p1.png (the original file is not changed)

The crop removes 80 pixels at the top, and the turn is 180 degrees. Without preparation, OCR gets 1/9 correct, with the wrong vendor Docu n. With it, OCR gets 9/9 correct in about 3 seconds. The vision model got 9 of 9 on the raw picture, in 10.25 seconds, for about US$0.0133.

See it end to endRead real receiptsCase study

The receipt case study measures preparation on 100 real receipt photos. There the raw photo wins: 146 correct values against 127 on the test receipts, because the crop sometimes keeps the wrong part of a photo.

Knowledge check

Check your understanding

Name the four preparation steps in their order, and say for each one how the program decides whether to apply it.

Show the answer

Crop: to the paper on a photo, or to the printed part of a white page. Turn: the program scores a quick OCR of a small copy (mean score × share of wide boxes). Under 0.85, it tries the other quarter turns and keeps a clearly better one. Straighten: if the long lines slope more than 1 degree. Enlarge: if the text is under 24 pixels high, to about 32, by at most 2.5 times. Every step is written down.

Check your understanding

Predict and explain: Tomás wants to switch preparation on only for photos, 'because photos are the messiest pictures'. What do the measurements say?

Show the answer

The opposite. Photos did not gain (103 of 117 printed values raw, 102 prepared): their problems are blur, cut-off parts and unknown words, not geometry. The large gains were on scans. Turned pages went from 0 to 8 of 8 and from 1 to 9 of 9; small receipts on big scanner pages from 71 to 79 of 85. Decide by the measurement per condition, not by how messy a picture looks.

Check your understanding

Practical check: a box from OCR on a prepared picture must be stored. What must happen first, and why must it point at the original file?

Show the answer

Undo the recorded steps in reverse order: scale, deskew, turn, then the crop (prepare.to_original). The original is the file that agents, reviewers and auditors can open, so the evidence must point at it. On D025, TOTAL at (11, 574) on the prepared picture becomes (1770, 1008) on the original page.

Module recap

This module took Larkfield’s files from the door to text with evidence. You can now stop bad files with a reason, and find near duplicates by content (Input handling). You can read a page in three ways into one normalised representation, and choose the path by the page (Text and OCR paths). And you can prepare pictures, measure the effect and keep every box on the original (this lesson). Your project matches the snapshot end-of-m02: python -m pytest -q prints 63 passed.

Module practice: two approaches on the same page

Inspect the outputs of two extraction approaches on the same sample, and explain every difference before any model is asked to fix it.

Guided: D045, raw and prepared

D045 is a phone photo whose bottom is cut off.

1. Compare the paths on the raw picture:

   python -m doc_extract.compare D045

   OCR gets 9/9 correct; the model gets 8/9 correct: it misses the tax 46.66. The total and the payment are cut off, and both paths rightly leave them empty.

2. Compare them on the prepared picture:

   python -m doc_extract.compare D045 --prepare

   OCR now gets 8/9 correct: the currency is missing.

3. Find the cause in the OCR lines. Run python -m doc_extract.ocr D045 | head -5 and python -m doc_extract.ocr D045 --prepare | head -5 (on Windows, use Select-Object -First 5 instead of head -5). The raw line is Toronto ON M5H 2M5; the prepared line is TorontoON M5H 2M5.

4. Write down each difference and its cause. The prepared OCR lost a space, so the rule that infers CAD from a Canadian address fails. The model missed a printed tax. Nobody invented the cut-off total.

Independent: D047, the photo with its top cut off

Now do the same alone on D047, a photo whose top is cut off. Run python -m doc_extract.ocr D047 and python -m doc_extract.compare D047. Which fields are not on the photo, and what does each path do with them? Which OCR lines would pair a label with the wrong amount if you read them from top to bottom? Would you send this receipt to a model?

Show a worked solution

The vendor and the date are cut off: the photo starts at Order: LK-192531. Both paths leave them empty, which compare marks ok: nobody invented a shop or a date. Both get 9 of 9.

Sorted from top to bottom, the OCR lines put each amount above its label. On this photo, the amounts on the right sit a little higher than their labels:

  1.00 [   869    500   1019    548]  187.00
  0.99 [   151    522    349    568]  Subtotal
  1.00 [   894    555   1020    602]  24.31
  0.98 [   152    575    330    622]  HST 13%
  0.97 [   779    608   1021    655]  CAD 211.31
  1.00 [   156    631    286    674]  TOTAL

Read naively, Subtotal is followed by 24.31 and HST 13% by CAD 211.31: two wrong pairs. The project’s rows follow the page’s slope and pair Subtotal with 187.00 and TOTAL with CAD 211.31. A model given these lines in this order could pair them wrongly too.

A model cannot read what the photo does not show, so a model call cannot find the vendor or the date. The honest answer is “absent, cut off”, and a person asks the customer for a better photo. The course’s pipeline does call the model on D047, because two required fields are missing. In Handle visual uncertainty, you see that the model then gives the wrong reason, and how the pipeline catches it.

Completion check: errors before the model

Identify reading-order and OCR errors before you apply a language model. A model that gets bad text cannot know which letters were misread, and it may “fix” them into plausible, wrong values. For each output below, list the errors. Say whether each one is an OCR error or a reading-order error, and what the project does about it.

1. The raw OCR of D025, at the start of this lesson.
2. The lines of D036 sorted from top to bottom, from Text and OCR paths: Sous-total, TPS 5 %, 880,00, TVQ 9,975 %, 44,00, TOTAL, 87,78, 1 011,78 $.

Show a worked solution

D025, raw OCR. OCR errors: IARILD (LARKFIELD), Homr eaden (Home & Garden), Dat00: 20-13 (the date), Oord 359 (the order number), CA 3.65 (CAD 38.65), Subttal, PS%, and a Chinese character for a dashed line. You can see the problem without the answer. 27 of 28 lines score under 0.80, and the boxes are taller than wide, so the text runs up the page. The cause is the page’s direction. prepare.py turns the page and gets 26 lines at 0.90 or more, and 9 of 9 fields. A model given the raw lines could turn CA 3.65 into a plausible, wrong CAD 3.65.

D036, top to bottom. Each line is read right: the error is the reading order. The photo is tilted by about 5 degrees, so the order pairs TPS 5 % with 880,00 (the subtotal) and TOTAL with 87,78 (a tax). pages.rows groups lines along the slope and gets the right pairs. Elsewhere on the page, OCR made one real error: Nettoyeur haute pressionPW-2200 lost a space.

In both cases, check the scores, the shape of the boxes and the order of the rows before any model. Fix what the reader can fix, and send only what is left to a model or a person.

Recap

• Image preparation works on a copy. It crops to the document, turns it upright after a quick OCR test, straightens a slope over 1 degree (deskew) and enlarges small text. Every step is written down.
• Turned pages gain the most: D025 from 0 of 8 printed values to 8 of 8, D026 from 1 of 9 to 9 of 9. Small receipts on big scanner pages gain from the crop. Plain, crooked, faded and low-resolution scans were already right. Photos do not gain (103 to 102 of 117).
• Over all 54 receipt pages, OCR goes from 392 to 408 of 435 printed values, for about 0.5 seconds more per page. Measure each change by condition: on D045, preparation lost a space, and so the currency.
• Keep the original. Every box goes back onto the file that the customer sent, so that anybody can check it.

Next, Module 3 recovers the structure of longer documents: sections, lists and footnotes in a policy with two columns.

Next: Sections and hierarchy