Not All Prefills Are Equal: PPD Disaggregation for Multi-turn LLM Serving

Prefill-Decode (PD) disaggregation has become the standard architecture for modern LLM inference engines, which alleviates the interference of two distinctive workloads. With the growing demand for multi-turn interactions in chatbots and agentic systems, we re-examined PD in this case and found two fundamental inefficiencies: (1) every turn requires prefilling the new prompt and response from the last turn, and (2) repeated KV transfers between prefill and decode nodes saturate the bandwidth, leading to high latency and even service degradation. Our key insight is that not all prefill operations are equally disruptive: append-prefill---processing only the new input tokens while reusing cached KV states---incurs substantially less decoding slowdown than full prefill. This motivates routing append-prefill to decode nodes locally. However, through comprehensive analysis, we show that no single fixed routing strategy satisfies all Service Level Objectives (SLOs) simultaneously. Based on this insight, we propose Prefill Prefill-capable Decode (PPD) disaggregation, a dynamic routing system that decides when to process Turn 2+ requests locally on decode nodes using cached KV states. PPD adapts to varying SLOs via configurable weights and seamlessly integrates with traditional PD deployments. With extensive evaluations, we show that PPD reduces Turn 2+ time-to-first-token (TTFT) by 68% while maintaining competitive time-per-output-token (TPOT), effectively alleviating KV transfer congestion under high load. We believe PPD represents a flexible and efficient paradigm for multi-turn LLM serving.

Modern AI chatbots and agents handle conversations with many back-and-forth turns. With current systems, every new turn forces the AI to re-process the entire prior conversation from scratch, which is computationally wasteful and creates communication bottlenecks between the specialized computing units that work together to serve each request. We investigated this problem and uncovered a key insight: not all parts of this re-processing are equally disruptive—certain "incremental" updates cause only ~2% slowdown to other ongoing work, versus ~48% for full re-processing. Building on this, we designed PPD, a smart routing system that dynamically decides where each part of a conversation should be executed. PPD cuts the time a user waits for the AI to start responding by up to 68%, making large-scale chatbot and agent deployments noticeably faster and more efficient.