From Model Metrics to System Behaviour: Measuring Reliability in Non-Deterministic AI

This talk presents a practical framework for evaluating increasingly complex, non-deterministic AI systems, moving from individual models to complete production pipelines. I begin with a page-stream segmentation model, where standard metrics concealed important failure modes at Wisedocs. For example, we discovered that the accuracy of identifying a document boundary is not the same as the ability to reconstruct the complete document span. Furthemore, errors on some document types carried greater downstream risk than others, and small aggregate improvements did not necessarily translate into meaningful business outcomes. This case study shows how evaluation for us evolved towards task-specific metrics, sensitive failure slices, strict regression testing, and more business-facing metrics of reliability.

I then move to LLM-based systems using Medical Long-Context Reasoning, or MLCR, as a case study. This was a benchmark we developed at Wisedocs earlier this summer. I will examine how to evaluate systems such as agentic chatbots and AI insight-generation services when outputs are open-ended, variable across runs, and difficult to score deterministically. I also explore the limitations of LLM-as-a-judge approaches, including judge disagreement, hallucination, inconsistent standards between judges, and the resulting need to evaluate the evaluators themselves.

Finally, I will consider Pipeline V2: a production system composed of multiple machine-learning models, LLM-powered components, and agentic workflows. I will discuss why component-level accuracy cannot be averaged into system-level reliability, how failures propagate across stages, and how end-to-end evaluation, observability, latency, cost, recovery, and customer outcomes must be combined to determine whether the overall system is successful.

The central lesson is that evaluation must grow with the system. As the unit of intelligence expands from a model to an AI application to a system of systems, benchmarks too must evolve to assess the distribution of real-world behaviour.