DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing

Modern LLM serving systems must sustain high throughput while meeting strict latency SLOs across two distinct inference phases: compute-intensive prefill and memory-bound decode phases. Existing approaches either (1) aggregate both phases on shared GPUs, leading to interference between prefill and decode phases, which degrades Time-Between-Tokens (TBT); or (2) disaggregate the two phases across GPUs, improving latency but wasting resources through duplicated models and KV cache transfers. We present DuetServe, a unified LLM serving framework that achieves disaggregation-level isolation within a single GPU. DuetServe operates in aggregated mode by default and dynamically activates SM-level GPU spatial multiplexing when TBT degradation is predicted. Its key idea is to decouple prefill and decode execution only when needed through fine-grained, adaptive SM partitioning that provides phase isolation only when contention threatens latency service level objectives. DuetServe integrates (1) an attention-aware roofline model to forecast iteration latency, (2) a partitioning optimizer that selects the optimal SM split to maximize throughput under TBT constraints, and (3) an interruption-free execution engine that eliminates CPU–GPU synchronization overhead. Evaluations show that DuetServe improves total throughput by up to 1.3$\times$ while maintaining low generation latency compared to state-of-the-art frameworks.

When you chat with an AI assistant, a powerful computer chip handles your request in two different steps: first it reads your whole question at once, then writes the answer one word at a time. The first step is like a sprint, the second a steady jog—and when one chip does both for many users at once, the two trip over each other, making replies stall and stutter. Today's systems either tolerate this interference or split the steps across separate chips, which stays smooth but leaves costly hardware idle. We built DuetServe, which lets a single chip run both steps at once without them colliding. It divides the chip's processing power into two lanes, but only when it predicts a slowdown coming, otherwise running at full efficiency. In tests, DuetServe handled up to 30% more work on the same hardware while keeping responses fast.