There's No One There to Press Retry: What We Learned Running AI on a Million Documents a Day

Everyone building on frontier models hits the same wall: 80% of the way there in a weekend, then an exponentially expensive climb toward the 99%+ that operational systems need. This talk is the honest map of that climb, from a team that now runs AI over millions of documents, images and videos a month for customers in logistics, fleet management, automotive and financial services who need the answer to be right every time.

We will walk through the ladder nobody budgets for: retries, then queues when the model is down for three hours, then rate limits, then discovering that 95% is not enough for transactional data, then discovering that the model cannot tell you how confident it is. We will show two failures from our own production history, a customer whose single engineer racked up $30K of usage in a month because nothing was watching, and our first churn, on 500-page reports with a hundred rows per page, a problem we still consider unsolved. And we will show what we changed: a harness that treats AI as a deterministic step inside durable workflows rather than as an open-ended agent, decisions expressed as a verified tree the model must traverse, algorithmic confidence scoring on top of models that provide none, routing anything under 95% to a human whose verdict feeds back into the system, semantic checks against instructions smuggled into the data, and, counterintuitively, encouraging customers to build their own independent monitoring of us. One customer’s users went from eight to nine hours a week on a task to about thirty minutes, and that number was measured by them, not by us.

Attendees will leave with five patterns they can apply Monday: confidence scoring for models that lack it, decision trees over open-ended prompts, exception routing with feedback loops, semantic security checks on inbound data, and customer-owned evaluation. They will also leave with a thesis we did not start with: chat is single-player AI; the next decade of software is ambient AI that runs the same process a hundred thousand times a day, unattended, and behaves the same way every time.