Letting AI Agents Run Incident Response on a 12M-Device Network: Guardrails, Gating, and 70% Lower Cost

We redesigned frontline incident response around AI agents that do not just suggest fixes but execute them, across more than 12 million devices and tens of thousands of incidents a month. Structurally, three things changed. First, agents stopped getting broad human-equivalent access and instead call tools through a governed interface, with permissions scoped down to individual functions and parameter values, so a compromised or confused agent has a bounded blast radius. Second, human approval moved from a blanket gate to a targeted one, applied only to sensitive or irreversible actions and routed by how familiar the problem is and how reversible the action is. Third, we stopped paying full model-inference cost for repeated work by promoting an agent’s proven, validated behavior into deterministic playbooks that run at near-zero token cost, which cut agent running cost by more than 70 percent over eight months while incident volume doubled.

Attendees will leave able to decide what an agent should be allowed to do at all, where a human actually adds signal versus just latency, and how to drive the cost of autonomy down over time instead of letting it grow. I will be specific about what broke first and the guardrails we only added after an incident.