A leaderboard win is not a product. Nemotron Retriever’s models rank #1 on RTEB — but topping a benchmark and running efficiently inside a customer’s production are two very different problems. This talk is the engineering story of closing that gap.

I’ll walk through our journey rebuilding the Retriever inference stack in Rust with a single obsession: treating state-of-the-art performance as a first-class product requirement, not a post-hoc optimization. We’ll get concrete about the decisions that mattered — what we measured, where we spent effort, the throughput, memory, and footprint wins we chased, and how dramatic efficiency gains unlocked deployment scenarios (including edge and on-device) that simply weren’t possible before. I’ll also share what didn’t work, the tension between shipping fast and building durable, and how we leaned on AI-assisted development to move faster than the roadmap assumed.

You’ll leave with a mental model for treating inference performance as a product and go-to-market lever — and a practical playbook for taking research-grade models to production without leaving speed, cost, or reach on the table.

Key takeaways:

1. Why “performance is the product”: efficiency is an adoption and go-to-market lever, not a cost center.
2. Concrete decisions from a Rust-based inference rebuild — what to measure, where to optimize, and how to know it worked.
3. How latency and footprint reductions open entirely new deployment surfaces (edge, on-device) for the same models.
4. Navigating the research-to-production seam: turning SoTA benchmark results into reliable, cheap, fast serving customers can bet on.

Who should attend / level? AI engineers, ML platform/infrastructure engineers, and engineering leaders who build, deploy, or operate inference and serving systems. Intermediate.

We redesigned distributed GPU orchestration for RL post-training and batch inference in the open-source llm-d platform. Structurally, we replaced static GPU/TPU locking with a three-tier co-operative time-slicing system:

• Application Layer: Workloads signal phase boundaries (rollouts, training, batch inference) via explicit acquire() and yield() APIs.
• Cluster Orchestrator: Manages lock queues to dynamically interleave complementary jobs onto shared hardware during idle phases.
• Node Snapshot Agent: Executes fast sub-second state swaps between GPU/TPU VRAM and host DRAM, enabling instant context switching without container restarts.

Attendees will walk away with:

• Drive 70%+ GPU/TPU Utilization: Understand how time-slicing reclaims idle hardware during RL loops and batch inference without impacting convergence.
• Architect Rapid Memory Swapping: Apply VRAM-to-DRAM snapshotting strategies for ultra-fast GPU/TPU context switching.
• Deploy on Kubernetes: Configure llm-d and K8s orchestrators to interleave RL and batch inference on shared clusters.

The ranking and recommendation systems landscape is being transformed in the generative era. This talk reports on the experience of building and shipping the training infrastructure behind the first generative recommender in production at Meta, covering both stages of the recipe: pre-training to acquire the generative capability and post-training RL to align generation with the ranking objective.

Generative recommenders bring distinct challenges to end-to-end performance and scalability. This primarily arises from a mixed architecture that consists of both recommender-native large sparse embedding tables and LLM-based decoders. This then generates semantic IDs translating to real-world use cases in production such as finding the right advertisement for a given user. We inherit the communication profile of a sparse recommender as well as the autoregressive nature of a large language model for which an end-to-end systems blueprint simply does not exist.

For pre-training, we make the significant change from discriminative to generative recommendation. We talk about feature processing, data loading and the user-modeling path while building the LLM decoder and semantic ID tokenization on top while addressing real-world productionization challenges and end-to-end LLM performance analysis.

Reinforcement learning was implemented as an extensible framework where rewards, losses, RL algorithms, reference models and generation strategies are pluggable. We’ll walk through the reward design and the end-to-end post-training pipeline that prioritizes scalability at the production scale at Meta.

We’ll close with end-to-end optimization work – redundant computation elimination, specialized kernels for very short sequences, dense-sparse pipelining and hybrid embedding placement that satisfies large-scale performance requirements in compute and throughput.

We believe the key output from this session is leaving attendees with a crisp understanding of generative recommendation systems in large-scale production. We want to ensure we stay practical and address real-world production challenges that are relevant to developers and builders in AI infrastructure today. We do this by diving into the technicalities of enabling a large-scale generative retrieval system as well as concrete performance optimizations that enable the shift from a research prototype to a highly-optimized production system critical to revenue. We also feel the training systems and infrastructure-side of LLMs in production is rarely addressed at least relative to model architecture and quality, and we hope to fill that gap with this session.