MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving

The surge of large language model (LLM) applications on personal devices imposes massive, bursty workloads on cloud serving infrastructure. While prefill-decode disaggregation improves throughput and scalability, memory-bound decode instances often suffer from persistent load imbalance, as output lengths are unknown when requests arrive at the cloud. To address this, we propose MAPS, a Memory-Aware Predictive Scheduling framework tailored for disaggregated LLM serving. MAPS performs device-assisted speculative output length prediction overlapped with cloud-side prefilling, incurring negligible latency overhead. To handle generation uncertainty, MAPS applies uncertainty-aware calibration to derive output-length upper bounds with target coverage, enabling safe scheduling decisions. Building on these bounds, MAPS employs a hierarchical global-local scheduling strategy to mitigate inter-decoder queue buildup and intra-decoder head-of-line blocking. Extensive experiments on two real-world workloads and two LLMs show that MAPS significantly outperforms three state-of-the-art systems, reducing average end-to-end latency by 42.6% and tail latency by up to 84.8%.

Millions of people now chat with AI assistants like ChatGPT every day, but behind every conversation, the AI runs on powerful servers in data centers that handle thousands of users at once. A key challenge is that the AI's responses can vary wildly in length: some answers take ten words, others ten thousand. Today's systems route incoming questions to servers without knowing how long the response will be, so some servers get overwhelmed with long answers while others sit idle. Users end up waiting much longer than necessary, especially during busy periods. We built MAPS, a system that estimates how long each AI response will be before it starts generating one. We do this by running a small, lightweight AI model on the user's own device while the main server is still preparing the answer, so the estimate is essentially free. MAPS then uses this estimate to send each request to the server most likely to finish it quickly, while shorter requests get prioritized to avoid being stuck behind long ones. In our experiments, MAPS cuts the longest waiting times users experience by up to 84%, making AI assistants feel noticeably more responsive without requiring any additional hardware. As AI assistants become part of everyday life, this kind of smarter scheduling helps ensure that AI feels fast and reliable for everyone.