Closing the Loop in the Agentic Software Development Lifecycle with Evals

Building agentic software requires more than adding an LLM to the traditional software development lifecycle. When behavior becomes non-deterministic, teams need a new feedback loop connecting what happens in production to how they evaluate, debug, and improve their systems. This talk will show how to build that loop: identifying real failure modes from production traces and human feedback, turning them into reproducible datasets and evaluations, defining success using both technical and business outcomes, and using automated and human judges to measure whether changes actually improve the system. We’ll also explore where automated evaluation breaks down, how to calibrate LLM-as-a-judge, and how continuous evaluation helps teams detect regressions and emerging behavior. Attendees will leave with a practical framework for moving from production > failure discovery > datasets > evals > iteration > production, turning the development of agentic systems into a measurable engineering discipline.