ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving

Large language model (LLM) serving demands low latency and high throughput, but high load variability leads to significant GPU under-utilization. In this paper, we identify a synergistic but overlooked opportunity to co-serve latency-critical online requests alongside *latency-tolerant offline* tasks, which existing systems fail to exploit because their coarse-grained resource management introduces interference. We present ConServe, a co-serving system that enables fine-grained resource sharing through latency-aware token-level scheduling, sub-iteration layer-wise preemption, and incremental KV-cache management. These mechanisms allow offline execution to fill *millisecond-scale* GPU idle time while preserving strict online latency guarantees. Across real-world workloads with Llama-3.1 and Qwen-2.5 models, ConServe improves throughput by 2.2$\times$ on average and reduces online tail latency by 2.9$\times$ over state-of-the-art systems.

Running a popular AI assistant is surprisingly wasteful. Traffic to these services can triple within seconds, so providers keep enough GPUs running to handle the biggest spikes, which leaves much of that hardware idle the rest of the time. The obvious fix is to use those idle moments for other work, but live users notice even a small added delay, and current systems cannot stop an AI model mid-response quickly enough to avoid that delay. ConServe is a serving system that solves this by managing GPU time at a much finer granularity than existing systems. It runs latency-sensitive user requests alongside "offline" jobs such as document summarization and model evaluations, using three pieces that keep the two from interfering: a scheduler that decides how much extra work fits inside the time budget for the next user response, a mechanism that pauses background work between the model's internal layers within a few milliseconds when a spike arrives, and a memory scheme that records only the small amount of new state generated at each step so paused jobs can resume without redoing earlier work. On Llama and Qwen models with real workloads, Concerto does $2.2\times$ more useful work on the same GPUs and makes worst-case user responses $2.9\times$ faster, lowering the cost and energy of running large AI models at scale.