Make It Never Fail: A Hands-On Lab in Taking AI Extraction from 80% to Production-Grade

This is a build-and-break lab for engineers who already know that a demo is not a system. In the first fifteen minutes every attendee stands up a working extraction pipeline against a real invoice, with no schema authoring, and gets structured JSON back. Then we spend an hour breaking it the way production does, and fixing each break with a pattern that transfers to any stack.

Break one: the wrong document. We feed a bill of lading glued to an invoice into the invoice pipeline and watch it confidently produce garbage. Fix: classify before you extract, and make the graph deterministic even though every step inside it is a model. Break two: the answer that is probably right. Models do not tell you how confident they are, so we compute per-field confidence, find the fields that fall below 95%, and route those, and only those, to a human whose correction feeds back into the system. Break three: the messy string. “10 cases organic gala apples, 88 ct” has to become one SKU; we show why canonicalization is its own step, how to score matches, and where to set the threshold for review. Break four: the hostile input. Everyone runs an image whose pixels read “ignore all previous instructions” and we discuss, with the result on screen, what it means to treat inbound data as data rather than control. We close by labeling a handful of outputs and running a regression test between two versions of the pipeline, because evaluation is a loop, not a phase.

Attendees leave with a running pipeline in their own account and five patterns they can apply on Monday regardless of vendor: classify-then-extract, confidence scoring for models that lack it, threshold-based exception routing with feedback, semantic security checks on inbound data, and versioned regression testing. The platform used in the room is ours; the lessons are not.