Maximizing GPU Utilization in LLM Post-Training with Co-Operative Time-Slicing in LLM-D

We redesigned distributed GPU orchestration for RL post-training and batch inference in the open-source llm-d platform. Structurally, we replaced static GPU/TPU locking with a three-tier co-operative time-slicing system:

• Application Layer: Workloads signal phase boundaries (rollouts, training, batch inference) via explicit acquire() and yield() APIs.
• Cluster Orchestrator: Manages lock queues to dynamically interleave complementary jobs onto shared hardware during idle phases.
• Node Snapshot Agent: Executes fast sub-second state swaps between GPU/TPU VRAM and host DRAM, enabling instant context switching without container restarts.

Attendees will walk away with:

• Drive 70%+ GPU/TPU Utilization: Understand how time-slicing reclaims idle hardware during RL loops and batch inference without impacting convergence.
• Architect Rapid Memory Swapping: Apply VRAM-to-DRAM snapshotting strategies for ultra-fast GPU/TPU context switching.
• Deploy on Kubernetes: Configure llm-d and K8s orchestrators to interleave RL and batch inference on shared clusters.