OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration

Serving Large Language Models (LLMs) can benefit immensely from parallelizing both the model and input requests across multiple devices, but incoming workloads exhibit substantial spatial and temporal heterogeneity. Spatially, workloads comprise heterogeneous requests with varying compute and memory demands. Temporally, workload composition varies over time. Nevertheless, existing systems typically assume spatially uniform and temporally stable workloads, employing a homogeneous, static model deployment. This mismatch between the assumption and real-world spatial-temporal heterogeneity results in suboptimal performance. We present OServe, an LLM serving system with heterogeneous and flexible model deployment that addresses both spatial and temporal heterogeneity. First, OServe introduces a novel workload-aware scheduling algorithm that optimizes heterogeneous model deployments according to real-time workload characteristics. Second, OServe proposes an efficient workload-adaptive switching method that migrates model deployments in response to predicted workload changes. Experiments on real-world traces show that OServe improves performance by up to 2$\times$ (average: 1.5$\times$) compared to state-of-the-art serving systems.

Large language models power tools such as chatbots, coding assistants, and document summarizers, but running them efficiently is difficult because user requests vary widely. Some requests are short and computation-heavy, while others produce long outputs and require more memory. The mix of requests also changes over time, but existing serving systems often use a fixed setup. This paper introduces OServe, a system that adapts large language model serving to changing workloads. OServe predicts future request patterns, chooses suitable resource allocations and parallelism strategies for different model replicas, and routes different workloads to the most suitable replicas. It can also switch deployment layouts quickly by transferring model parameters and KV cache across high-bandwidth GPU connections, rather than reloading the full model from scratch. On real-world traces, OServe improves serving performance by up to 2$\times$, with an average improvement of 1.5$\times$.