Fan-Out, Tail Latency, and Cluster Mutations: Three Problems in An Ad-Serving Feature Store at Hundreds of Nodes

Instacart’s ad-serving feature store served features to real-time inference from a multi-hundred-node managed Valkey deployment split across multiple clusters. The team had built a proxy layer, a querying SDK, and a compact storage format on top of it, and still hit three limits: instability during any cluster mutation, tail latency from read fan-out, and cost that got worse when they split clusters to manage the first two.

The structural change was to stop scaling out and scale up. A multi-threaded engine let the team run much larger instances, so hundreds of nodes became roughly 100, each inference request touched far fewer shards, and average and P99 latency fell 50%. Because the proxy and SDK already hid the datastore from ML engineers, the swap required no application changes and tens of terabytes moved without anyone above the storage layer noticing.

We were the engine vendor, and the first pass did not reach the latency the team expected. Getting there took client-side tuning on their end and engine changes from our engineering team, including to the compactor. Validation ran side by side: identical backfill, traffic shifted from 0% to 100%, gates at each percentile.

Attendees will leave able to test a datastore against the fan-out shape of their own feature store rather than an ops/sec number, trace how engine concurrency sets instance size, which sets fan-out, which sets P99, and run a datastore migration under live traffic with per-percentile gates and a rollback path. They will also hear where the trade-off comes back as the cluster grows.