Not Every Production Alert Needs an Agent

An on-call agent swarm multiplies the cost of production alert noise. They happily burn tokens investigating alerts and coming up with plausible sounding, but incorrect root causes. To mitigate this requires understanding their biases and an alert triage layer in front of the agent swarm.

Per alert, that layer answers three questions: is this worth investigating, does it belong with other alerts as one incident, and how many agents does it get? It feeds co-firing alerts into the swarms context and throttles agent parallelization.

We evaluated the ability of agents to reach the same root cause humans achieved on incident post-mortems. We quickly found that there’s a lot of context that agents didn’t have access to or didn’t consider investigating. Understanding the context that is most helpful to identifying a root cause also allowed us to better guide agent task ownership. Through iterative refinements we improved their performance.

Identifying the production alerts that provide the greatest signal-to-noise allowed us to deploy the swarm with precision. Having humans review the swarm’s conclusions provides new training data to optimize their behavior with.

You’ll leave with a playbook for deploying agents for on-call alert triaging, mistakes we made that we hope you won’t repeat, and an orchestration layer that keeps agent cost bounded.