Threshold-Based Exclusive Batching for LLM Inference

Mixed batching (MB)—interleaving prefill and decode in a single batch—has become the standard scheduling strategy for large language model (LLM) inference due to its efficiency in maximizing compute and memory utilization. However, through controlled experiments, we find that prefill–decode interference inflates MB's per-step marginal cost above that of pure decode. On the high-bandwidth H200 (4.8 TB/s), this occurs only when decode tokens exceed 80% of the batch; however, on the bandwidth-constrained RTX PRO 6000 (1.792 TB/s), this threshold plummets to just 20%. Consequently, the optimal choice between MB and exclusive batching (EB) fundamentally depends on GPU memory bandwidth, model size, and workload composition. We derive a closed-form condition for this EB–MB performance crossover, along with asymptotically optimal phase-switching thresholds and memory-safe batch sizing for EB. Optimized EB achieves up to 41.9% higher throughput on bandwidth-constrained GPUs, while MB retains its advantage on high-bandwidth hardware with larger models. Our hybrid scheduler EB+ applies this condition online to dynamically switch between EB and MB without manual intervention. Under non-stationary traffic with distribution or concurrency shifts, EB+ attains the highest or near-highest throughput in every setting, outperforming MB by up to 36.4%.

When you ask an AI chatbot a question, the underlying system processes your request in two distinct phases: first, it reads your entire prompt at once to understand the context, and second, it generates the reply word by word. To handle a massive influx of user requests efficiently, state-of-the-art AI inference engines "mix" these two types of jobs together in the same processing batch (referred to as mixed batching, or MB, in our paper). This mixed approach has become the Western industry standard. However, an intriguing geographical divide has emerged: many large-scale production systems in China deliberately avoid this, preferring to keep the two jobs strictly separated (referred to as exclusive batching, or EB). Why the disagreement? Our research reveals that the answer largely lies in the hardware—specifically, a GPU's memory bandwidth. On high-bandwidth chips, mixing the two jobs works seamlessly. But on chips with tighter memory constraints—such as those widely used outside the U.S. due to export restrictions—mixing the jobs creates a computational "traffic jam" that severely degrades performance. To resolve this, we developed a mathematical model that precisely dictates the optimal moments to switch between these jobs for EB, and established a formula predicting exactly which strategy will be faster for a given chip, model, and workload. Using this math, we designed a hybrid "smart scheduler" (EB+) that adaptively shifts between MB and EB on the fly without requiring manual intervention. Ultimately, it delivers up to 41.9% higher throughput on memory-constrained GPUs while perfectly matching the efficiency of the industry standard on high-end hardware.