Little's Law Meets the Roofline Model: A Decision Framework for Sizing Hosted LLM Inference

We will start with two diagnostic tools: Little’s Law which translates your QPS and target latency into the concurrency and batch size your system actually needs and the roofline model which tells you whether prefill (compute-bound) or decode (memory-bound) is your real constraint before you reach for a fix..

Will walk through the levers that move the needle most – MoE vs. dense architecture and what that does to your communication pattern FP8/INT4 quantization and where it costs you accuracy, TP/EP/DP parallelism and when each is the right tool and batching to the B_sat knee before compute-bound latency hockey-sticks..

Will cover concrete tuning playbooks for two workload types:

1. latency-sensitive traffic like chat and agents where you’re optimizing TTFT and P99 ITL with chunked prefill and modest batch sizes
2. Throughput-sensitive traffic like batch summarization and evals where you push past B_sat and let queuing work in your favor.

Most production systems live in both worlds at once, so we’ll need a balanced configuration to tackle both.

You will leave with a 7-step decision framework – characterize your workload, pick your model and quantization, benchmark on candidate hardware, find the knee, size your deployment with Little’s Law, calculate total cost of ownership, and plan for autoscaling that you can apply to your own inference stack..