WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving

Deploying multiple models within shared GPU clusters is a key strategy to improve resource efficiency in large language model (LLM) serving. Existing multi-LLM serving systems improve GPU utilization at the cost of degraded inference performance, particularly time-to-first-token (TTFT). We attribute this degradation to the lack of awareness regarding future workload characteristics. In contrast, recent analyses have shown the strong periodicity and long-term predictability of real-world LLM serving workloads. In this paper, we propose \textit{one-for-many GPU prewarming}, which proactively loads parameters from multiple models onto GPUs based on workload forecasts. These prewarmed weights enable the system to promptly instantiate serving instances upon encountering request bursts. We design and implement WarmServe, a multi-LLM serving system incorporating three key techniques: (1) a model placement algorithm that optimizes prewarming decisions to minimize cross-model prewarming interference, (2) a KV cache reservation strategy that repurposes idle KV cache space on running GPUs for prewarming new models, and (3) an efficient GPU memory switching mechanism for tensor management. Evaluation on real-world datasets shows that WarmServe reduces tail TTFT by up to 50.8$\times$ compared to the state-of-the-art autoscaling-based system, while supporting up to 2.5$\times$ higher request throughput than the GPU-sharing system.

Modern AI platforms host dozens of specialized large language models — some answer chat questions, others write code, and others tackle complex reasoning. Cocurrently running all of them efficiently is expensive because each one demands powerful, scarce GPU hardware while user demand swings wildly throughout the day. Today's systems face an uncomfortable trade-off: keeping every model permanently loaded wastes hardware during quiet periods, but loading models on demand forces users to wait many seconds when traffic suddenly spikes. Sharing one GPU among several active models avoids the wait but cripples each model's working memory and hurts performance. We start from a simple observation: although minute-to-minute traffic looks chaotic, daily and weekly usage patterns are remarkably predictable. We use these forecasts to "prewarm" GPUs — loading the weights of several likely-needed models in advance, so a serving instance can be activated almost instantly when a burst arrives. We further developed techniques that decide which models to prewarm together, sneak preparations into GPUs that are about to free up, and switch between models without disruption. Our system, WarmServe, responds up to 50× faster to traffic spikes and serves up to 2.5× more requests than today's leading systems, making AI services cheaper to run and faster for users.