When the input to your pipeline is a PDF someone emailed
Invoices arrive as email attachments and someone retypes them. Doing that well is less about extraction accuracy than about the documents the model gets wrong.
A logistics client's accounts team starts the day the same way every day. Someone opens a shared mailbox, works down the unread list, opens each attachment, and retypes what is in it into the ERP. Invoice number, supplier, date, line items, quantities, unit prices, tax, total. Enough of them on a normal Monday to fill two people's morning, and the same again the next day.
Nobody designed this. It accumulated. The supplier portal that was meant to replace it never got adopted because half the suppliers are small firms with an accountant who emails a PDF and considers the matter closed. So the pipeline that feeds the finance warehouse, the one with the tests and the lineage and the orchestration graph, begins with two people and a keyboard.
This is the most common automation brief we see, and it is also the one most likely to be scoped badly. The brief usually arrives as "can AI read our invoices". The honest answer is yes, mostly, and that is not the interesting part. The interesting part is what happens to the documents it reads badly, because that is what decides whether the system saves anyone any time.
Accuracy is the wrong headline number
Extraction accuracy is easy to quote and easy to misread. Suppose a model gets most fields right on a test set. Sounds good. Now put it in front of a clerk who does not know which ones are wrong. They have to check every document, which means they are now doing the original job plus reading the model's suggestions. You have added work.
The number that matters is different. It is the proportion of documents that can pass through with nobody looking at them, at an error rate the business has already agreed it can live with. Call it the straight-through rate. Everything in the design serves that one figure, and the second half of the sentence is not optional: an error rate finance will accept for a delivery note is not the error rate they will accept for a payment run.
A system that is usually right and cannot tell you which answers are the wrong ones saves nobody any time at all.
The pattern, stated plainly
The shape of an intake that works
Every document intake we have built that survived contact with a real back office has the same four parts. The parts are not novel. The discipline is in refusing to skip any of them.
- 01Capture and quarantineThe mailbox is an ingestion surface, not a workspace. Every attachment lands in object storage with its sender, subject, receipt time and a content hash. The hash kills the duplicate problem, which is larger than anyone expects once you count resends and forwarded chains. Nothing downstream ever reads the mailbox again.
- 02Extract to a schemaThe model is asked for a defined structure, not for prose. One field per thing you actually need, with types. Ask for the supplier's tax identifier as a string with a stated format, not for "any reference numbers on the document". Also ask, per field, where on the page it came from, so a reviewer can be shown the evidence rather than asked to hunt for it.
- 03Validate before you trustDeterministic checks run on the extracted object. Do the line items sum to the subtotal. Does the subtotal plus tax equal the stated total. Does the supplier resolve to a row in the vendor master. Is the purchase order real and open. Is the date inside a sane window. This layer catches things no confidence score will, because it is checking against the business, not against the pixels.
- 04Route on the outcomeClean and confident goes straight through. Anything that fails a check, or where the model is unsure, or where the value crosses a materiality threshold, goes to a review queue. The routing rules are configuration, owned by finance, not constants buried in a function.
The review queue is the product
Teams building this tend to treat the review screen as the consolation prize, the bit for the failures. It is the opposite. It is the only surface a human touches, so it determines whether the whole thing is felt as a relief or a nuisance, and it is where your evaluation data comes from for free.
A good one puts the document image on one side and the extracted fields on the other, highlights the region a field was pulled from, focuses the cursor on the first field that failed validation, and makes the common correction a single keystroke. It never asks someone to retype a field the model got right. If a reviewer finds themselves fixing every field on the document, the routing has failed upstream and you should be logging that as a defect, not as review volume.
Now the useful part. Every correction is a labelled example: the document, what the model said, what a person who knows the business said instead. Write those to Postgres alongside the extraction, and within a few weeks you have a held-out set drawn entirely from your own traffic, weighted by how often each failure actually happens. That set is worth more than any public benchmark, because it contains your suppliers, your layouts, your handwriting, your smudged fax from the depot.
Evaluate the way you route
With that set in hand, evaluation stops being vague. Score per field, not per document, because a wrong tax code and a wrong delivery address are not the same event. Score the routing decision separately from the extraction: a document that was wrong and got sent to review is a success, a document that was wrong and went straight through is the only real failure. Keep the two numbers apart and you can tune the threshold deliberately instead of arguing about whether the model is good.
- Field-level exact match, weighted Weight each field by what a mistake costs. Getting the bank account wrong is not the same class of error as getting the delivery note reference wrong.
- Straight-through rate Share of documents with no human contact. This is the money number and it should be tracked per supplier, because a handful of suppliers usually account for most of the queue.
- Escaped error rate Errors that passed the gate. Sampled by pulling a fixed number of auto-approved documents each week and checking them properly.
- Time on the exception Median seconds a reviewer spends per queued document. Rising time means the review screen is fighting them, even if accuracy is flat.
Run this as an eval harness in the pipeline, not as a spreadsheet someone updates when they remember. Prompt changes, schema changes and model upgrades all get scored against the same set before they go anywhere near the mailbox. That is the point at which the AI part becomes ordinary engineering.
Where this is the wrong answer
Three situations where we would tell you not to build it, and have.
If the volume is a trickle, the payback is not there. A short daily stint of typing costs less than a system that will need attention forever. Spend the budget on chasing your largest few suppliers onto a structured feed instead, which is the real fix and is often just an email asking for a CSV. Structured beats extracted every time, and one supplier moved off PDFs is worth more than any amount of prompt tuning.
If the document drives an irreversible action with no downstream check, be careful. Extracting a payment instruction and executing it is a different risk category from extracting an invoice that will still pass through approval and a bank confirmation. Where nothing catches the mistake afterwards, the review gate has to be much wider, and the economics may stop working.
And if nobody owns the vendor master, stop. Half of what looks like extraction failure is actually reconciliation failure: the model read the supplier name perfectly and there are four spellings of that supplier in the ERP. No amount of prompt work fixes that. Clean the reference data first, and the extraction problem gets noticeably smaller on its own.
What is left, when those three are out of the way, is a genuinely good use of the technology. Not because the model is clever, though it is. Because the alternative is two people retyping a mailbox full of PDFs before lunch, and neither of them ever wanted that job.