Document-processing agent on Azure AI Foundry

Problem#

Finance received supplier documents as scans, e-mail attachments and portal exports in a dozen layouts. A template-based extraction tool covered the four largest suppliers and nothing else, so roughly 70% of documents were keyed by hand, twice, for four-eyes control.

Architecture#

Ingestion drops documents in Blob Storage and raises an event. An Azure AI Foundry agent orchestrates the pipeline: layout analysis, field extraction, validation against master data, and a confidence-scored result. Low-confidence fields route to a Power Apps review screen; approved results post to the ERP through a queued Azure Function.

Every run stores its input hash, prompt version, model version and output, which makes any posted document reproducible after the fact.

Solution#

Rather than one large prompt, extraction is split per field group with narrow schemas and strict JSON validation. Anything that fails validation is never silently repaired. It is sent to review with the failing rule attached.

The review screen was designed with the two people who actually do the work; it shows the document and the extracted field side by side and takes a keyboard-only path from first field to submit.

Results#

• 94%: Fields auto-approved
• 60k: Documents per year in scope
• 11 sec: Median end-to-end processing time

Lessons learned#

Confidence thresholds are a product decision, not a model setting. We tuned them with finance until the review queue was a size a human could actually clear in a morning.