Performance is the Product: Our Journey to SoTA for Nemotron Retriever

A leaderboard win is not a product. Nemotron Retriever’s models rank #1 on RTEB — but topping a benchmark and running efficiently inside a customer’s production are two very different problems. This talk is the engineering story of closing that gap.

I’ll walk through our journey rebuilding the Retriever inference stack in Rust with a single obsession: treating state-of-the-art performance as a first-class product requirement, not a post-hoc optimization. We’ll get concrete about the decisions that mattered — what we measured, where we spent effort, the throughput, memory, and footprint wins we chased, and how dramatic efficiency gains unlocked deployment scenarios (including edge and on-device) that simply weren’t possible before. I’ll also share what didn’t work, the tension between shipping fast and building durable, and how we leaned on AI-assisted development to move faster than the roadmap assumed.

You’ll leave with a mental model for treating inference performance as a product and go-to-market lever — and a practical playbook for taking research-grade models to production without leaving speed, cost, or reach on the table.

Key takeaways:

1. Why “performance is the product”: efficiency is an adoption and go-to-market lever, not a cost center.
2. Concrete decisions from a Rust-based inference rebuild — what to measure, where to optimize, and how to know it worked.
3. How latency and footprint reductions open entirely new deployment surfaces (edge, on-device) for the same models.
4. Navigating the research-to-production seam: turning SoTA benchmark results into reliable, cheap, fast serving customers can bet on.

Who should attend / level? AI engineers, ML platform/infrastructure engineers, and engineering leaders who build, deploy, or operate inference and serving systems. Intermediate.