Operational constraints

What you will be able to do

Know what a page costs before the bill arrives: seconds, tokens, money, retries, memory and storage, for each way to read it. Multiply these unit costs by the volume you expect, and write the assumptions down. Then Omar’s question, “what does it cost per page?”, has a number and its reasons.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The time and cost of each path on one page, from Text and OCR paths.
• Why the combined pipeline calls the model on 9 of 54 documents only, from Combine methods.
• Tokens and list prices, from Cost model in AI System Design and Cost Engineering, if you took it. This lesson explains each calculation.

What you need on your computer

• Your project at the end of the previous lesson: python -m pytest -q prints 101 passed.
• No key and no money. The model’s tokens and times come from the course’s recorded calls.

Tested with Python 3.12.13 and RapidOCR 3.10.0 on macOS (Apple silicon). The Windows and Linux commands are not tested. Times on your computer will differ: compare the ratios, not the seconds.

Get the lesson’s file

1. Copy the module and its test from the snapshot:

   macOS / Linux

   S=../nextia-learning-labs/C13/snapshots/end-of-m06
   cp $S/doc_extract/budget.py doc_extract/
   cp $S/tests/test_budget.py tests/

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C13\snapshots\end-of-m06"
   Copy-Item "$S\doc_extract\budget.py" doc_extract\
   Copy-Item "$S\tests\test_budget.py" tests\

2. Run the tests:

   python -m pytest -q

   The last line is 103 passed.

Plan a road trip

Before a long drive, you make a small budget. Your car uses about 7 litres of fuel per 100 km. You know this number because you measured it on earlier trips. The trip is about 1,200 km, but that is a plan, and detours happen. So you add a margin. You also check two limits that money does not fix: how many hours you can drive in one day, and what fits in the boot.

The budget has two kinds of numbers. Some are measured: the fuel per 100 km. Some are assumptions: the distance and the detours. You multiply them, and you write down which is which. When the trip costs more, you know which number was wrong.

The problem: “just send every page to the model”

Omar, the product manager, asks Mei the question that every manager asks: “What does it cost per page?” Tomás has an answer ready: “The vision model reads 97.7% of the values right. Send every page to it. It is one cent a page.”

One cent sounds like nothing. But nobody has multiplied it yet. Larkfield’s help desk may get 300 pages a day. The model’s median answer takes 7.35 seconds, and one recorded call took 132 seconds. The answer also depends on what runs the OCR. One OCR process, with the engine’s default settings, grew to 2,180 MB of memory on the course’s pages (measured). A worker with a 2 GB memory limit stops in the middle of a batch.

None of these limits shows in an accuracy table. Each one decides if the pipeline works on a busy day.

The idea: unit costs times volume, with the assumptions written down

An operational budget lists what one page costs on each path, then multiplies by the expected volume. It has five parts:

• Time per page: how long a page takes on each path. It decides how many workers you need and how long an agent waits.
• Tokens and money per page: what the model reads and writes. Tokens are the pieces of text that a model provider counts and bills. A picture is billed as tokens too.
• Retries: some calls fail or time out, and you send them again. Each retry is another paid call.
• Memory per worker: the largest amount of memory that one process needs, its peak. It decides how many workers fit on one machine.
• Storage per document: the original file and everything that the pipeline makes from it.

Each number gets a label. Measured means the course ran the project’s code and timed it. Recorded means a real model answer, stored with its tokens and time. An assumption is a number that you choose, such as the pages per day. The project’s budget command prints these labels.

A tiny example by hand

Take 300 pages a day, and assume that 25% of them need the model. Assume also that 2% of the model calls are sent again. Each number of the calculation is visible:

model calls per day  = 300 pages x 0.25 x 1.02           = 76.5 calls
tokens of one page   = 3,164 in + 556 out                  (recorded medians)
price of one page    = 3,164 x US$2 / 1,000,000            = US$0.0063  (input)
                     +   556 x US$10 / 1,000,000           = US$0.0056  (output)
                                                           = US$0.0119
money per day        = 76.5 x US$0.0119                    = US$0.91
OCR time per day     = 300 pages x 1.34 s                  = 402 s = 6.7 CPU minutes

Output tokens cost five times as much as input tokens, at the list price that the course uses. So the 556 tokens that the model writes cost almost as much as the 3,164 that it reads. This is why the precise prompt costs more than the plain one: its answer is longer.

A day of 300 pages. Dashed boxes are assumptions; green boxes are the results. Change an assumption, and the results follow.

Text description of the diagram

A flow of a day’s budget. 300 pages a day, an assumption, split in two. 75% go to the text layer or OCR and the rules, for no money. OCR takes 1.34 seconds a page on this computer: 6.7 CPU minutes a day. 25% go to the model: 75 calls, plus 2% repeated calls, which gives 76 calls. Each call costs US$0.0119, from the recorded median tokens (3,164 in, 556 out) at the list price. The result is US$0.91 a day, about US$27 a month.

Budget the course’s pipeline

1. Run the budget with the default assumptions:

   python -m doc_extract.budget

   Measured on this computer (sample of 6 documents):
     text layer   6 ms per page, US$0
     OCR          1.34 s per page (with preparation), US$0 (your CPU)
   From the course's 56 recorded vision calls (median):
     vision model 7.35 s per page, 3164 tokens in, 556 out = US$0.0119 per page at list price
   
   Assumptions: 300 pages a day, 25% of them sent to the model, 2% of model calls repeated
     model calls per day   76  -> US$0.91 a day, about US$27 a month
     OCR time per day      6.7 CPU minutes if every page needs OCR
     all pages to the model instead: US$3.64 a day
     storage: originals about 148 kB each (median); keep them, they are the evidence

   The command reads six documents with the text layer and OCR, without the cache, so it takes about 7 seconds. The OCR time changes from run to run: the course saw from 1.12 to 1.34 seconds a page, with other work running. It is also not the course’s median of 0.97 seconds over all 54 receipt pages. The budget times only a sample of six documents, at the moment you run it. Your time will differ. The model’s numbers do not change, because they come from the recordings.

2. Send every page to the model, as Tomás proposed:

   python -m doc_extract.budget --pages 300 --model-share 1.0

   Assumptions: 300 pages a day, 100% of them sent to the model, 2% of model calls repeated
     model calls per day   306  -> US$3.64 a day, about US$109 a month

   This is four times the money. It also adds 7 seconds of waiting to every page, also to the digital receipts that the text layer reads in 5 milliseconds.

3. Measure the real share of the combined pipeline. In python:

   from doc_extract import combine, documents
   docs = documents.receipts()
   used, cost = [], 0.0
   for d in docs:
       ex, dec = combine.combine(d)
       if dec.used_model:
           used.append(d.doc_id)
           cost += ex.usage.get("cost_usd") or 0
   print(f"the model read {len(used)} of {len(docs)} documents: {', '.join(used)}")
   print(f"US${cost:.3f} in all, US${cost / len(docs):.4f} per document")

   the model read 9 of 54 documents: D017, D018, D033, D034, D045, D046, D047, D048, D049
   US$0.097 in all, US$0.0018 per document

   Six of the nine are the other shop’s receipts, whose words the rules do not know. The other three are the damaged photos. Nine of 54 is 17%.

4. Put the measured share into the budget:

   python -m doc_extract.budget --pages 300 --model-share 0.17

   Assumptions: 300 pages a day, 17% of them sent to the model, 2% of model calls repeated
     model calls per day   52  -> US$0.62 a day, about US$19 a month

The share of 17% is measured on the course’s documents, not on Larkfield’s real mail. If more customers send receipts from partner shops, the share goes up. So the share stays an assumption in the budget, but now it has a reason.

• Assumed share of pages
• Share measured on the course documents

The model's cost of a day of 300 pages, for three shares of pages sent to the model, with 2% of calls repeated. From the budget command: recorded tokens at the list price, times assumed volumes.

Text description of the chart

The model’s cost for a day of 300 pages, with 2% of calls repeated. Every page to the model: US$3.64. A quarter of the pages: US$0.91. The share measured on the course’s documents, 17%: US$0.62.

Time: the slow path decides the wait

The four paths differ by a factor of more than a thousand. The course measured them on all 54 receipt pages:

• Runs on your CPU
• A call to the model (recorded)

Median seconds per page on the course's 54 receipt pages. The text layer and OCR ran on the course's computer; the vision model's times are the recorded calls.

Text description of the chart

Median seconds per page. The text layer: 0.0054 seconds. OCR on the raw picture: 0.49. OCR with preparation: 0.97. The vision model: 7.35.

A median hides the slow calls, and the slow calls make agents wait. The course looked at every recorded call of the precise prompt (a check over the whole set, run by the course):

54 receipt pages; tokens in: median 3135, min 1269, max 3194; tokens out: median 556
cost per page: median US$0.0113, max US$0.0143, all 54 pages US$0.60
seconds per page: median 7.35, p95 10.65, max 11.49
second pass (56 calls): max 132.2 s
  digital_pdf   21 pages, tokens in median   1431, cost median US$0.0098
  scan_pdf       7 pages, tokens in median   3194, cost median US$0.0124
  scan_image    10 pages, tokens in median   3194, cost median US$0.0114
  photo         15 pages, tokens in median   3135, cost median US$0.0115

The p95 is the time that 95% of the calls stay under: 10.65 seconds. When the course sent the same pages a second time, one call took 132 seconds. So a worker that calls the model needs a timeout: a time after which it stops waiting. Then it sends the call again, or sends the page to review. Each repeated call is paid again. That is the retry rate of the budget.

The narrow till receipts cost fewer tokens than the letter-size scans: 1,431 against 3,194, because the picture is smaller. The budget command takes the median over all 56 recorded calls, which also include two problem files. So it prices a page at US$0.0119, a little above the US$0.0113 of the receipt pages. For a budget, the higher number is the safe side.

Memory: how many workers fit on one machine

A worker that runs OCR keeps the OCR models and a large picture in memory. OCR works in two steps: a text finder finds where the lines of text are, then a reader reads each line. The text finder works on a copy of the page, made smaller to a maximum size. The course measured the peak memory of one process that read every receipt page:

• The engine's default size
• The project's setting

Peak memory of one process that reads the course's receipt pages with OCR, for two picture sizes of the text finder. Both settings read 408 of 435 printed values right. Measured on the course's computer.

Text description of the chart

Peak memory of one OCR process. With pictures up to 2,000 pixels, the engine’s default: 2,180 MB. With pictures up to 1,280 pixels, the project’s setting: 1,447 MB. Both read 408 of 435 printed values right.

With the default size, one process needs more than 2 GB. A limit of 1,280 pixels for the text finder’s picture saves about 730 MB. The course found no loss: the same 408 values are right. This is the project’s default, OCR_MAX_SIDE=1280. So the rule for a plan is one OCR worker per about 1.5 GB of memory. A server with 8 GB free runs at most five OCR workers, with some memory left for the system. Measure again when the documents change: in the Read real receipts case study, one process that prepared 100 real phone photos peaked at about 2.5 GB at the same setting, and about 1.25 GB on the raw photos.

Storage: keep the evidence

In python:

import statistics
from pathlib import Path
originals = [p.stat().st_size for p in Path("data/documents").iterdir() if p.is_file()]
cache = [p.stat().st_size for p in Path("out/cache/ocr").glob("*.json")]
print(f"originals: {len(originals)} files, {sum(originals) / 1e6:.1f} MB, median {statistics.median(originals) / 1e3:.0f} kB")
print(f"OCR cache: {len(cache)} pages, {sum(cache) / 1e6:.1f} MB, median {statistics.median(cache) / 1e3:.0f} kB")

originals: 65 files, 17.3 MB, median 131 kB
OCR cache: 110 pages, 0.4 MB, median 4 kB

Your cache count can differ: it depends on the commands that you ran. The originals are much larger than everything the pipeline makes from them. Keep them anyway: they are the evidence that every box points to. The cache can always be made again, at the cost of OCR time. Take 300 receipt files a day at the receipts’ median of 148 kB. Then the originals grow by about 44 MB a day, or 16 GB a year. That is an assumption times a measurement, and your retention rules decide how long you keep them.

When to budget, and what the budget cannot tell you

• Budget before you choose a path, with your own volumes. A path that is cheap per page can be expensive per month.
• Measure time on your own machine. The course’s times are one computer’s, with other work running. Keep the ratios: the model is about a thousand times slower than the text layer.
• Keep the labels. When the bill is higher than the budget, check the assumptions first: the share of pages sent to the model, and the retries.
• Prices change. The course uses a list price of US$2 per million input tokens and US$10 per million output tokens. Check the current price page of your provider.
• A budget is not a quality measure. The cheapest plan, rules only, cannot read the other shop’s receipts. Use the evaluation of the previous lesson to choose, then the budget to plan.

Common mistake

The OCR worker that stopped every night

Symptom: Every night, a batch of scans stops at about the same place. The job queue shows the same pages failing again and again. In the morning, the worker log ends without an error message.

Cause: The worker runs in a container with a 2 GB memory limit. Somebody set the text finder back to its default picture size of 2,000 pixels. On the course’s pages, one OCR process then reaches 2,180 MB, and the system stops it. Each retry starts the same pages and stops again. This is a constructed story, but the memory numbers are the course’s measurements.

Diagnose: Measure the peak memory of one worker on a real batch, with an empty OCR cache. On macOS, put /usr/bin/time -l before the worker’s command: it prints the “maximum resident set size” at the end. On Linux, use /usr/bin/time -v. Compare the peak with the container’s limit.

Fix: Keep OCR_MAX_SIDE=1280, the project’s default: 1,447 MB, with the same values right. Plan one OCR worker per about 1.5 GB. Stop the retries of a job after a few failures, and send its pages to review with a reason.

Consequence in AI work: models need memory as well as time and money. A setting that is better on one page can stop a whole batch, and a retry without a limit turns one failure into a loop.

Your turn: a bigger help desk

Change one condition: Larkfield buys a second shop, and the help desk expects 1,000 pages a day. Partner receipts are more frequent, so the calls fail more often: assume 10% of the model calls are repeated. Keep 25% of the pages to the model. Predict the model calls and the money per day. Then run:

python -m doc_extract.budget --pages 1000 --model-share 0.25 --retry-rate 0.10

Show the answer

Assumptions: 1000 pages a day, 25% of them sent to the model, 10% of model calls repeated
  model calls per day   275  -> US$3.27 a day, about US$98 a month
  OCR time per day      21.0 CPU minutes if every page needs OCR
  all pages to the model instead: US$13.08 a day

By hand: 1,000 × 0.25 × 1.10 = 275 calls, and 275 × US$0.0119 = US$3.27. The volume grew 3.3 times and the retry rate five times, but the money grew only 3.6 times. The retries are a small part of the bill. The OCR time is still small: 21 CPU minutes a day fit easily in one worker. With 10% retries, though, look at why calls fail before you pay for them. A timeout that is too short repeats calls that would have succeeded.

Knowledge check

Check your understanding

Explain why the budget command labels the pages per day and the model share as assumptions, but the tokens per page as recorded.

Show the answer

The tokens per page come from real calls to the model, stored with their counts. The course measured them once, and they do not change when you run the command. The pages per day and the model share are choices about the future. Nobody has measured Larkfield’s real mail yet. The labels tell the reader which numbers to check first when the bill differs from the budget: the assumptions.

Check your understanding

Predict: the team crops every receipt picture before the model call, and the input tokens of a page fall from 3,164 to 1,300. The output stays 556 tokens. What does a page cost now, at the course's list price?

Show the answer

Input: 1,300 × US$2 / 1,000,000 = US$0.0026. Output: 556 × US$10 / 1,000,000 = US$0.0056. Together US$0.0082 a page, about a third less than US$0.0119. The output does not change, so it is now the larger part of the cost. A crop cannot make a page cheaper than its answer.

Check your understanding

Practical check: run python -m doc_extract.budget --pages 300 --model-share 0.17. How many model calls a day does it print, and how many OCR workers would a server with 6 GB free run?

Show the answer

It prints model calls per day 52 -> US$0.62 a day: 300 × 0.17 × 1.02 = 52 calls. One OCR worker needs about 1.5 GB at the project’s setting. So 6 GB fit three workers, with some memory left for the system. One worker could already do the OCR of a day: about 6 to 7 CPU minutes for 300 pages. The extra workers help when many receipts arrive at once.

Recap

• A budget multiplies unit costs by volumes, and labels each number: measured, recorded or assumption.
• The model costs about US$0.0119 a page at the list price. Every page to the model: US$3.64 a day for 300 pages. The combined pipeline sends 9 of 54 documents: US$0.0018 per document.
• Time: 5 ms for the text layer, about 1 s for OCR, 7.35 s for the model, with slow calls up to 132 s. A worker needs a timeout, and each retry is paid.
• Memory: one OCR process needs 1,447 MB at the project’s setting, and 2,180 MB with the engine’s default. Plan one OCR worker per about 1.5 GB.
• Storage: keep the originals, which are the evidence. The cache can be made again.

Next, you run the pipeline in the background next to the ticket system. You reprocess after a change, delete a customer’s data, and guard each change with a regression set.

Next: Integration and maintenance