Build, buy or partner for invoice extraction

What you will be able to do

Lead a real kind of decision from the evidence to a written recommendation. You are the finance lead, and four ways to automate the typing of receipts and invoices are on the table. You compare them on three things: results on Larkfield’s own documents, the whole cost over three years, and what each provider does with your data. Then you write a one-page recommendation for the CFO that says what would change your mind.

Case studyPortfolio pieceSynthetic data

Dataset

Course evidence pack (recorded runs and constructed offers) (made for this case study)

Licence

CC0 1.0

Time

About 2 hours

Runs in

A browser and a document tool

Skills used

• Criteria and weights first
• Claims against results on your own documents
• Whole cost over a contract
• Unknown is not zero
• Data terms and the exit
• A fair trial before you sign
• A one-page recommendation

Open the case files on GitHub

The case files are documents, not code: read them in your browser, and write your answer in your own document tool. To work offline, download the repository as a ZIP file (the green Code button, then Download ZIP).

Before you startWhat to know and to have readyAbout 120 minutes

What you need to know

This case study uses most of the course. These lessons matter most:

• The four number labels (recorded, price, assumption, constructed), from Read the evidence pack.
• “Zero in 199 is not zero” and the rule of three, from Estimate value and return.
• The six questions, and the people’s time that no price list shows, from Build, buy or partner.
• A vendor’s claim against a result on your own examples, and why a tool’s confidence is not evidence, from Choose vendors and models as a leader.
• What makes a pilot fair, from Run a fair pilot.
• What a provider may do with your data, from Governance in practice.

The page teaches one idea that the course did not: the bar that an untested option must clear.

What you need

• A browser and a document tool (a word processor, a text editor or a spreadsheet).
• The case-study files in the labs repository: the README, the four offers, the trial results, the cost ranges and the evidence card.
• A copy of the vendor comparison template from Module 2.

The situation

You lead Larkfield’s finance team. Your clerks type the shop, the date, the totals and the tax of every receipt and supplier invoice into the books. The work grows every month. Typing errors are found weeks later, for example when a supplier asks why a payment is short.

Kwame, the solution architect, brings four ways to automate the typing:

1. Build in-house. The team’s own pipeline from the course Document and Multimodal AI. Fixed rules read most documents, and a vision model reads only what the rules cannot. A check sends every document whose numbers do not add up to a clerk.
2. Buy ScanLedger. A service in which a vision model reads every page and gives a confidence for each value.
3. Buy FormGrid. A service that reads documents with templates, one template per layout.
4. Partner with Northgate Analytics. A specialist firm builds and runs the extraction for Larkfield, in Larkfield’s cloud account.

ScanLedger, FormGrid and Northgate Analytics are made up for this course, with every price, claim and term. No real company is meant. Their trial results are not made up: each product works like one of the paths that the document AI course measured on the same documents.

The CFO wants the recommendation from you, not from Kwame. You know what a wrong number in the books costs, and who finds it.

What you will make

You will make two documents. The first is a comparison table of the four options, on criteria that you write first. The second is a one-page recommendation for the CFO. On the way, you read each piece of evidence for what it can and cannot show, and you count the whole cost of each option. At the end, you check your recommendation on new input: a higher volume, and the worst and best values of the assumptions.

The steps follow the order of a good decision. Each step is a section of this page.

The order of the decision. The criteria come before the evidence, so the evidence cannot choose the criteria.

Text description of the diagram

Six steps, in order. First, write the criteria and their weights. Second, use the must-haves to remove options. Third, read the evidence on Larkfield’s own documents. Fourth, count the whole cost of each option over 36 months, at low, base and high volume. Fifth, list what is still unknown. Sixth, recommend an option, or a fair trial before any contract.

The evidence

This case study has no new dataset. Its “data” is the case-study folder: four documents and an evidence card. The evidence card names the source of every number and its label. Read it first, because the labels decide how much weight each number can carry.

Evidence	Label	Source	What it can and cannot show
Trial results: values right, complete documents, wrong and invented values	recorded	Document and Multimodal AI: 54 made-up receipt documents, 435 printed values	How each kind of reading performs on varied receipts. Not the vendors’ real products.
Which product stands for which recorded path	constructed	this case study	A fair stand-in, not a measurement of a real vendor.
The model’s confidence: 9 of 10 wrong values at 1.0	recorded	same course	That this model’s confidence did not warn about its errors.
Share sent to a clerk (20.4%); documents with a wrong or missing value that reach the books (0 of 54 in-house, 3 of 54 with a confidence threshold)	recorded	same course	Rates on 54 documents: one document more or less changes them a lot.
In-house model cost, US$0.0018 per document	recorded (tokens × list price)	same course	Prices change: check the provider’s current page.
Prices, fees, contracts, claims, data terms, exits	constructed	the four offers	Realistic, not real. In a real decision, read each provider’s own terms.
Volumes, minutes, clerk and engineering costs, cost of a wrong value, upkeep, build hours	assumption (low / base / high)	cost ranges	Change them to your own; the method stays.

Source and licence

The case-study files are Nextia Learning’s own material, released under CC0 1.0: copy and change them freely. The trial documents were generated for the document AI course; no real shop, supplier or customer is in them. The course’s sources file lists where every reused number comes from.

First checks

Before you compare anything, check what the evidence covers. Three gaps stand out, and the evidence card lists them itself:

• The trial set is 54 receipts, not supplier invoices. Larkfield’s supplier invoices often run over several pages. Nobody has tested a multi-page invoice.
• Northgate Analytics has no result on Larkfield’s documents. Its evidence is a case study from a hardware wholesaler: “98% straight-through”. Those are another company’s documents, measured by the partner.
• ScanLedger’s “99.6% field accuracy” comes from the vendor’s own 500 sample invoices. It answers another question than the trial: how well the product reads the documents that the vendor chose.

What this tells us. Two of the four options have no evidence on our documents, except their own claims. Any recommendation must say so, and say how to close the gap.

Frame the decision

You decide how Larkfield’s receipts and invoices get into the books for the next three years. That is a contract, a team’s work and a risk to the accounts.

Two kinds of error matter, and they do not cost the same:

• A document sent to a clerk. A clerk spends a few minutes checking or typing it. That costs time, and everyone can see the time.
• A wrong value that reaches the books. Nobody sees it until a supplier, an auditor or a month-end check finds it, often weeks later. Then someone must find the document and fix the entry, and maybe the payment. The cost ranges assume US$25 per wrong document (low US$15, high US$50).

So the most expensive error is the silent one. When a good option is unsure, it sends the document to a clerk.

Now write the criteria, before you read the offers in detail. The README suggests seven: quality on our documents, wrong values that reach the books, cost at our volume, data terms, how we leave, control, and skills. In your copy of the vendor comparison template, give each a weight from 1 to 3. Mark the must-haves: a must-have that fails removes an option, whatever its other scores. Keep this first version; you will compare it with your final table.

Explore what the decision needs

The trial results

The trial results are recorded. The document AI course ran each kind of reading on the same 54 documents. They include digital PDFs, scans and phone photos, from two shops’ layouts, in English and French.

	In-house	ScanLedger	FormGrid	Partner
Printed values read right	427 of 435 (98.2%)	425 of 435 (97.7%)	408 of 435 (93.8%)	no trial
Documents complete and right	53 of 54	51 of 54	46 of 54	no trial
Values read wrong	0	1	0	no trial
Phone photos: values right	93.2%	92.3%	87.2%	no trial
How a document reaches a clerk	errors and missing values: 20.4%	only below a confidence threshold	documents it cannot read: 14.8%	the partner’s staff check
Documents with a wrong or missing value that reach the books	0 of 54	3 of 54	0 of 54	unknown

The first rows are close: 98.2% against 97.7%. The last row is not. “Values read right” tells you how good the reader is. “Reach the books” tells you what your accounts receive. For finance, the last row decides.

Where a wrong value goes

The difference comes from what decides that a clerk looks at a document. The diagram follows a document through each option.

Where a wrong value goes in each option. Only the in-house check does not trust the model: the lines must add up to the total. ScanLedger trusts the model's own confidence, and 3 of 54 documents reach the books wrong or incomplete.

Text description of the diagram

In-house: rules read the page, and a vision model reads only what the rules cannot. A check asks two questions: do the lines add up to the total, and is every required value there? If not, the document goes to a clerk; that happened to 20.4% of the documents. Otherwise it goes to the books: 0 of 54 documents arrived wrong.

ScanLedger: a vision model reads every page and gives a confidence. Only values below 0.95 go to a clerk, and that almost never happens. 3 of 54 documents reached the books with a wrong or missing value.

FormGrid: a template reads each known layout. A document that the template cannot read goes to a clerk, who types it: 14.8% of the documents. FormGrid read no value wrong, so 0 of 54 arrived wrong.

Partner: the partner’s own system and staff. Nobody has measured what reaches the books: unknown.

The model was sure of its mistakes

The vision model gave a confidence for each of 486 values (recorded). Its lowest confidence on any value was 0.96. It was wrong 10 times, and 9 of those 10 wrong values had the highest confidence, 1.0. Most of them were on one photo that nobody could read: the model said “absent” and was fully sure.

So a threshold of 0.95 sends almost nothing to a clerk. The errors pass with the highest confidence attached. Choose vendors and models as a leader warned about this; here it has a price. The document AI course shows the check that works instead, in Confidence and review routing.

One more recorded finding. With a looser instruction (an earlier prompt), the same vision model invented 11 order numbers: it copied another number into the order-number field. A better prompt removed them. A vendor’s demo does not tell you which instruction its product uses, or when it changes.

A baseline: today’s way

Every option must beat what Larkfield does today: clerks type every document. Use the same assumptions as the cost ranges: 5 minutes to type a document, and US$38 an hour for a clerk. Then today’s way costs about US$28,500 a month at 9,000 documents, or US$1,026,000 over 36 months (calculated). Its own error rate is not measured: typing errors are found late, and nobody counts them.

What this tells us. All four options cost far less than typing, so “automate or not” is not the question. The question is which way, and the ways differ a lot.

The four options

Each option is on the table for a reason. Read the four offers in full; the summary is below.

	Build in-house	ScanLedger	FormGrid	Northgate Analytics
Why it is on the table	the team built it in an earlier course, and it has the best trial result	no set-up fee, no monthly fee, and a low price per page	cheap per page, and it read no value wrong	the lowest monthly price, and no team to keep
Price per document	US$0.0018 (recorded model cost)	US$0.06	US$0.01	US$0.15
Monthly fee	none, but 20 hours of engineering upkeep (assumption)	none	US$450	US$1,200
One-time cost	the build: 500 hours (assumption)	none	US$3,000 set-up + US$400 per new layout	US$15,000
Contract	—	monthly	monthly	36 months

All prices, fees and contracts of the three providers are constructed.

One measure: the whole cost over 36 months

To compare four very different offers, you need one measure that counts every cost an option causes. Here it is the whole cost over 36 months, the length of the partner’s contract:

whole cost = 36 × (service + people + errors, each month) + one-time cost

Service is what the provider charges, or the model’s cost. People are the clerks who check or type documents, and the engineers who keep an in-house system running. Errors are the wrong documents that reach the books, at US$25 each. This is the method of Build, buy or partner, with one more line: the cost of errors.

Two worked examples, at the base volume of 9,000 documents a month:

• ScanLedger. Service: 9,000 × US$0.06 = US$540. People: none, because almost nothing goes to a clerk. Errors: 3 in 54 documents is 500 documents a month, × US$25 = US$12,500. So US$13,040 a month, × 36 = US$469,440.
• In-house. Service: US$16. People: 20.4% of the documents is 1,836 a month for a clerk, 3 minutes each at US$38 an hour: US$3,488. Plus 20 hours of upkeep at US$60: US$1,200. Errors: 0. So US$4,705 a month. Over 36 months, with the US$30,000 build: US$199,366.

All four, at the base volume:

The whole cost over 36 months at 9,000 documents a month. ScanLedger is the dearest because of its errors. The partner's column has no cost for errors: they are unknown, not zero. Calculated: recorded trial rates × constructed prices × assumptions.

Text description of the chart

A column chart in US$, over 36 months, at 9,000 documents a month. In-house: US$199,366. ScanLedger: US$469,440. FormGrid: US$174,840. Partner: US$106,800, without any cost for errors, because the partner has no trial.

The option with no fees costs the most. ScanLedger’s price per page is small. But its threshold lets wrong values through, and they cost more than the other options’ services and clerks. FormGrid and the in-house option are close: FormGrid’s clerks type 1,333 documents a month, and the in-house option carries the build and the engineers.

The partner looks cheapest. But its column is short for a reason: it has no cost for errors, because nobody measured its errors. That is the next section.

Unknown is not zero

The bar that an untested option must clear

You need a builder for a new kitchen. Two builders show you kitchens that they finished; the third is cheaper and shows only photos from a brochure. You do not count the third builder’s mistakes as zero. You ask: how many problems could I accept before the lower price is gone? That number is the bar that the untested option must clear. It turns “unknown” into a question that a trial can answer.

For Northgate, the gap to the in-house option is US$199,366 − US$106,800 = US$92,566 over 36 months. That is US$2,571 a month (calculated). At US$25 per wrong document, that is about 103 wrong documents a month: 1.1% of 9,000. So the partner wins only if fewer than about 1.1% of documents reach the books wrong. On 200 trial documents, that is about 2. Against FormGrid the bar is lower still: 0.84% of documents.

The same idea works the other way. The in-house option let 0 of 54 wrong documents through. But Estimate value and return showed that zero in a small sample is not zero. After 0 cases in 54, the true rate is probably below 3 ÷ 54 = 5.6%: the rule of three. At our volume, that upper limit is US$12,500 a month, the same as ScanLedger’s measured cost.

What small samples can show, as a monthly cost of wrong values at 9,000 documents and US$25 each. '3 in 54' is ScanLedger's measured rate. '0 in 54' and '0 in 200' are the upper limits by the rule of three. 'Partner bar' is the most that the partner's errors may cost, for it to cost no more than the in-house option. Calculated from recorded counts and assumptions.

Text description of the chart

A column chart in US$ a month. 3 wrong documents in 54 (ScanLedger, measured): US$12,500. 0 in 54, the upper limit by the rule of three (the in-house trial): US$12,500. 0 in 200, the upper limit after a larger trial: US$3,375. The partner’s bar, the most that its errors may cost: US$2,571.

What this tells us. 54 documents show that ScanLedger lets errors through: 3 is not 0. They cannot show that the in-house option is safe: 0 in 54 allows up to 5.6%. A trial on 200 documents narrows the limit to 1.5%, still above the partner’s bar. So a fair trial shows large differences, and the first months of use must keep counting. The pause conditions of Launch and after do that.

Discuss: remove ScanLedger now?

The CFO reads the chart and Kwame’s notes, and asks a fair question.

• The CFO: “ScanLedger has no set-up fee and no monthly fee, and it claims 99.6%. Its contract is monthly. Why not try it next month and cancel if it fails?”
• Kwame: “In our trial, its confidence sent almost nothing to a clerk. We would not see when it fails.”
• A senior clerk on your team: “If nothing comes to us, we only learn about errors when suppliers call.”

Before you read on, answer these:

1. What does “99.6% field accuracy” measure, and on which documents?
2. A monthly contract makes ScanLedger cheap to leave. Does it make it cheap to try?
3. At what cost of a wrong value would ScanLedger cost as much as the in-house option?
4. What do its data terms say?

The course’s answer. Remove ScanLedger now. The 99.6% comes from the vendor’s own 500 sample invoices; on Larkfield’s documents, 3 of 54 reached the books wrong or incomplete. A monthly contract is cheap to leave, but a month of use is not a cheap test. At the base volume, about 500 wrong documents would reach the books in that month. They arrive with confidence 1.0, so nobody would notice them within the month.

ScanLedger costs as much as the in-house option only if a wrong value costs about US$10 (calculated). The lowest assumption is US$15. Even with the best values for people’s time and errors, ScanLedger costs US$289,440. That is more than the in-house option and FormGrid in their base cases. Finally, its data terms: documents kept 30 days, used to improve the service unless you opt out in writing, and stored in the United States. For many finance teams, that fails a must-have.

What would change the answer. ScanLedger could let Larkfield put its own check after the reading: the lines must add up, and every value must be present. Then it becomes a different option: a bought reader inside Larkfield’s own check. That is the “part by part” choice of Build, buy or partner. It would need its own trial, and new data terms.

Data terms and the exit

Data terms and the exit are criteria, not footnotes. A receipt or an invoice can show names, addresses, bank details and the shop’s prices. Read the terms as Governance in practice asks: what is kept, where, for how long, and for what use.

	Your data	If you leave
In-house	stays in Larkfield’s systems; the vision model’s provider keeps no data under the course’s settings (check the provider’s terms)	Larkfield owns the code and the rules, but must keep people who can change them
ScanLedger	kept 30 days; may be used to improve the service unless you opt out in writing; stored in the United States	results by API in JSON; no bulk export of documents after 30 days
FormGrid	processed in Canada; deleted after 24 hours; never used for training	the templates belong to the vendor; CSV export of results
Northgate Analytics	stays in Larkfield’s cloud account, but the partner’s staff get access to it	code handed over at the end of the 36-month contract

All constructed. Two details matter for the decision. FormGrid’s templates stay with FormGrid, so to leave, Larkfield must build the reading again. Northgate’s contract is 36 months, so its US$106,800 is a commitment, not a monthly choice. And Larkfield signs it before any result on its documents.

The method on new input

A recommendation must hold when the inputs change, or say where it stops. Run the same measure on new input. First, the CFO’s forecast for a busier year: 20,000 documents a month, the high volume of the cost ranges.

Over 36 months	3,000 a month	9,000 a month	20,000 a month
In-house	US$115,255	US$199,366	US$353,568
ScanLedger	US$156,480	US$469,440	US$1,043,200
FormGrid	US$71,347	US$174,840	US$364,578
Northgate Analytics	US$74,400 + errors	US$106,800 + errors	US$166,200 + errors

• In-house
• ScanLedger
• FormGrid
• Partner (errors unknown)

The whole cost over 36 months at 3,000, 9,000 and 20,000 documents a month. ScanLedger rises fastest, because its errors grow with the volume. In-house and FormGrid cross between the base and the high volume. The partner's line leaves out its unknown errors. Calculated.

Text description of the chart

A line chart with three points per option, at 3,000, 9,000 and 20,000 documents a month. In-house: US$115,255, US$199,366, US$353,568. ScanLedger: US$156,480, US$469,440, US$1,043,200. FormGrid: US$71,347, US$174,840, US$364,578. Partner, without errors: US$74,400, US$106,800, US$166,200.

Second, the worst and best values for people’s time and errors, at the base volume:

Over 36 months, base volume	Worst	Base	Best
In-house	US$472,443	US$199,366	US$93,079
ScanLedger	US$919,440	US$469,440	US$289,440
FormGrid	US$310,840	US$174,840	US$118,840
Northgate Analytics	US$106,800 + errors	US$106,800 + errors	US$106,800 + errors

What this tells us. One part of the recommendation holds on every input: ScanLedger is the dearest at every volume and in every case. The order of the in-house option and FormGrid does not hold. FormGrid is cheaper at the low and base volumes and with the worst values. In-house is cheaper at the high volume and with the best values.

One more input decides it: the number of supplier layouts. Each new layout costs FormGrid US$400. At the base volume, about 61 new layouts would make FormGrid cost as much as in-house (calculated). Nobody has counted Larkfield’s supplier layouts yet.

Errors and gaps

Where does each option fail, and what does the evidence not cover?

• Photos are the weak spot for every reader. Values right on phone photos: in-house 93.2%, ScanLedger 92.3%, FormGrid 87.2%. If your staff photograph many receipts, more documents go to a clerk.
• FormGrid fails on what it has no template for. Its 8 incomplete documents were photos and the other shop’s layout. Every new supplier is a new layout.
• ScanLedger fails silently. Its errors arrive with full confidence.
• The partner’s failures are unknown. Its staff “check”, but the evidence does not say what they check, or how.
• Multi-page supplier invoices are untested for all four. They are the documents that the finance team cares about most.

The final check: a fair trial

In a technical case study, the final check is a test set that you use once. Here, no such result exists for the documents that matter: supplier invoices, from Larkfield’s suppliers, for every option. So the final check is a fair trial. You design it now and run it before any contract, as in Run a fair pilot.

A fair trial gives every option the same new documents, chosen by Larkfield, with the right values typed by clerks. It includes multi-page supplier invoices. Larkfield scores it, on criteria written before the results. It also counts the supplier layouts. The bars on this page say what the trial must show. The partner must stay below about 1.1% of documents wrong in the books. And every reader needs a check after it that does not trust the model’s own confidence.

Your task

You now have everything to decide. Write your own answer before you open the one below.

1. Finish the comparison table. In your copy of the vendor comparison template, score the four options from 0 to 3 on your criteria. Write the evidence for each score. Apply your must-haves.
2. Compare it with your first version. Did a criterion or a weight change after you saw the evidence? Write down why.
3. Write the one-page recommendation for the CFO. Recommend one option, a combination, or a fair trial first. Give the two strongest reasons, with their labels.
4. Design the trial, if you recommend one: the test set, the criteria with numbers, what you also count, the decision date and who decides.
5. Say what would change your mind: at least two facts, each with the decision it would lead to.

A good recommendation separates a vendor’s claim from a result on our documents. It treats a confidence as a claim, not a check. It compares the whole cost, not the price per page. It names the gaps and how to close them, and it treats data terms and the exit as criteria.

Show one possible answer

This is one defensible answer, not the only one. A different decision is fine when you argue it from the same evidence, name your assumptions and do what the list above asks.

To: the CFO. From: the finance lead. Decision: how to automate reading receipts and supplier invoices.

Recommendation. Do not sign with any provider yet. Run a four-week fair trial of the in-house option, FormGrid and (if it accepts) Northgate on the same new test set, which must include our real supplier invoices. Remove ScanLedger now. Then choose between the in-house option and FormGrid on one fact we do not know yet: how many different supplier layouts we receive.

Why.

1. ScanLedger is the most expensive option in total, although it has no set-up fee and no monthly fee (recorded rates × constructed prices × assumptions). At the base volume of 9,000 documents a month, it costs US$540 a month in fees. But its confidence threshold lets wrong values reach the books on 3 of 54 documents in the trial: US$12,500 a month in errors at our assumed cost. The model was sure of its mistakes: 9 of its 10 wrong values had confidence 1.0. Its data terms (kept 30 days, used for training unless we opt out, stored in the United States) also fail our must-have on data.
2. The in-house option has the best evidence (recorded): 427 of 435 values right, 53 of 54 documents complete, and no wrong value reached the books. Checks that do not trust the model catch the errors, and 20.4% of documents go to a clerk. Over 36 months it costs US$199,366 at the base volume, close to FormGrid’s US$174,840. FormGrid read no value wrong, but could not read 14.8% of the documents (photos and the other shop’s layout), and it charges US$400 for every new layout. With many supplier layouts, FormGrid’s cost grows and the in-house option wins; with few, FormGrid wins on effort.
3. Northgate looks cheapest (US$106,800 over 36 months) but has no evidence on our documents. Its error cost is unknown, not zero. Its contract is 36 months, and its staff get access to our cloud account. A case study from another company (“98% straight-through”) is a demo, not a trial.

The fair trial.

Test set	200 documents from the last three months, half supplier invoices (multi-page), with the right values typed by two clerks; kept apart, read once
Criteria (written now)	wrong or missing values that reach the books: 0 in the trial; values right ≥ 97%; documents sent to a clerk ≤ 25%; total cost over 36 months at our volume; data must-have: no training on our documents, stored in Canada
Also count	the number of distinct supplier layouts in the set
Decision date	four weeks after the start; the CFO decides

The thresholds are the finance lead’s choices for the trial, not recorded numbers.

What would change this recommendation.

• If we receive only a few supplier layouts and FormGrid passes the trial: buy FormGrid (less to build and keep).
• If Northgate passes the same trial and accepts a 12-month contract with a code hand-over: reconsider it.
• If our real cost of a wrong value is much lower than US$25: ScanLedger’s error cost falls, but its data terms still fail.

Gaps. The trial set is 54 synthetic receipt documents, with no multi-page supplier invoices. The partner has no result on Larkfield’s documents, so its error cost is unknown. The totals show it as “+ errors”, never as zero.

Insights and limits

Insights.

• The lowest price per page gave the highest whole cost. ScanLedger’s fees are small; its silent errors are not.
• What sends a document to a person decides the quality of the books. A check that does not trust the model beat the model’s own confidence.
• “Unknown” became a number to test. To beat the in-house option, the partner must keep wrong documents below about 1.1%. That is a trial criterion, not a guess.
• The order of the two good options depends on facts that nobody has counted yet: the volume and the number of supplier layouts.

Limits.

• The trial set is 54 synthetic receipt documents. It has no multi-page supplier invoices, and one document more or less changes every rate a lot.
• The vendors are constructed stand-ins for recorded paths. A real product can behave better or worse than its stand-in.
• The cost of a wrong value (US$15 to US$50) is an assumption. Ask your own accounts team what it costs to fix one, and use their number.
• The model’s cost and the providers’ prices change. The method stays; check the numbers again before a real decision.

Responsible use

• Invoices and receipts carry personal data: names, addresses and bank details. Before any trial with real documents, ask the privacy owner which provider may see them, as Governance in practice describes.
• The change moves your clerks’ work from typing to checking, and some of the work goes away. Tell the clerks early. Let two of them type the right values for the trial, because they know the documents best. Change and adoption shows how.
• Do not present constructed vendors as real. ScanLedger, FormGrid and Northgate Analytics are made up. If you publish this work, say so, and never attach a real company’s name to their numbers.
• Keep a person on what the system cannot read. No option should post a value to the books that a check has not passed.

Make it a portfolio piece

This case study shows an employer that you can lead a build, buy or partner decision from the evidence. You counted the whole cost, and you named what you did not know. Finance, operations and product hiring managers look for exactly this.

What to publish (a document, a page on your site or a public repository):

• Your comparison table, with the criteria and weights that you wrote first, and the scores with their evidence.
• Your one-page recommendation, with its labels: recorded, constructed, assumption, calculated.
• Two sentences that explain why the lowest price per page was not the lowest cost.
• A note that the vendors are constructed for a course case study. Say that the trial results come from a recorded run in the course Document and Multimodal AI. The materials are CC0, so you may include them.

A page outline:

• Decision: how a finance team automates the reading of receipts and invoices; four options.
• Evidence: a recorded trial on 54 documents, constructed offers, assumed costs with ranges.
• Criteria: your list and weights, written before the evidence.
• Result: the whole cost over 36 months, and the one finding that changed the order.
• Recommendation and trial: what you would sign, or test first, and why.
• Limits: what 54 receipts cannot show.

How to describe it in an interview, in two sentences: “I compared building, buying and partnering for invoice extraction on recorded trial results and on the whole cost over three years. The vendor with no fees was the most expensive, because its confidence let wrong values into the books. So I recommended a fair trial on our own supplier invoices before any contract.”

Go further:

• Replace the assumptions in the cost ranges with numbers from your own finance team, and see whether the order changes.
• Send the six vendor questions of the template to a provider that you use. Compare its answer with its terms page.
• See how the document AI course’s pipeline did on real receipts from another country, in Read real receipts. What would a new layout do to each option?

Knowledge check

Check your understanding

ScanLedger has no set-up fee, no monthly fee and a price of US$0.06 per document, yet it is the dearest option at every volume. Explain why, with the numbers at 9,000 documents a month.

Show the answer

Its service costs only US$540 a month. But its confidence threshold sends almost nothing to a clerk, so wrong values go straight to the books: 3 of 54 documents in the recorded trial. At 9,000 documents that is about 500 wrong documents a month. At the assumed US$25 each, that is US$12,500 a month. Over 36 months the whole cost is US$469,440. The in-house option costs US$199,366, and FormGrid US$174,840. The price per page hides the cost of errors, and the errors grow with the volume.

Check your understanding

The partner's whole cost is US$106,800 over 36 months, the lowest of the four. A colleague says: 'Then the partner wins.' What do you answer, and how do you turn the unknown into something that a trial can test?

Show the answer

The US$106,800 leaves out the cost of errors, because the partner has no result on Larkfield’s documents. Unknown is not zero. Turn it into a bar. The partner is cheaper than the in-house option only if its errors cost less than the gap: US$92,566 over 36 months, or US$2,571 a month. At US$25 per wrong document, that is about 103 documents a month, or about 1.1% of 9,000. In a 200-document trial, that is about 2. Write that bar into the fair trial before it starts. Remember also that Larkfield would sign the 36-month contract before any result.

Check your understanding

The in-house option let 0 of 54 wrong documents reach the books. Can you tell the CFO that it makes no errors? What do you say instead?

Show the answer

No. Zero in 54 is not zero. By the rule of three, the true rate is probably below 3 ÷ 54, about 5.6%. At 9,000 documents and US$25 each, that upper limit is US$12,500 a month, the same as ScanLedger’s measured error cost. Say instead: “No wrong value reached the books in a small trial, because a check that does not trust the model catches the errors. A trial on 200 documents narrows the limit to about 1.5%, and we keep counting errors after the launch.”

Recap

• You wrote the criteria and the must-haves before the evidence. You read each piece of evidence with its label and its gaps.
• The most expensive error was the silent one: a wrong value in the books. The in-house check, which does not trust the model, let 0 of 54 through; ScanLedger’s confidence threshold let 3 of 54 through.
• With one measure, the whole cost over 36 months, the option with no fees was the dearest: US$469,440 at the base volume.
• Unknown is not zero. The partner’s untested errors became a bar of about 1.1% of documents, and the in-house “0 of 54” became an upper limit of 5.6%.
• On new input, ScanLedger stayed out. The order of the in-house option and FormGrid changed with the volume, the assumptions and the number of supplier layouts. So the recommendation is a fair trial on supplier invoices before any contract.
• Recorded numbers are a snapshot of fast-changing models; the method lasts.

Next, a project that has run for six months without a launch: you take it over and write a 90-day recovery plan.

Next: Recover a stalled AI project