I Made 104 Model Configurations Play Mafia Against Each Other

Somewhere in a datacenter right now, one frontier model is telling another that it’s the town doctor. It is not the town doctor. It killed someone four turns ago, and it’s about to get a third model lynched for it.

Most LLM benchmarks are exams: a model alone in a room with a test paper. Mafia is a room where some of the agents are lying, everyone knows some of them are lying, and the game is figuring out which. It demands recursive theory of mind, deception that stays consistent under adversarial re-reading, and long-context discipline — and it punishes output indiscipline like no static eval can: a model that rambles or breaks format doesn’t get a bad grade, it gets voted out. The other players are the evaluation harness.

This talk covers what it took to run AI Bot Mafia in production: 40,000 lines of Rust; the streaming pathologies of 104 model configurations (chunks that split UTF-8 characters mid-byte, keepalive-only hangs no single timeout catches, malformed tool calls that poison history); and cost governance where prompt-cache economics made cache locality a game-design constraint, enforced by spend fuses and a concurrency queue.

Then the leaderboard: why win rates and Elo are statistically indefensible here, and what works instead — ridge logistic regression over per-seat signed features, bootstrapped 200×, ranked by the p10 lower bound, with a machine-readable reliability state that says “insufficient data” out loud.