AugServe: Adaptive Request Scheduling for Augmented Large Language Model Inference Serving

Augmented large language models (LLMs) that invoke external calls are increasingly prevalent in inference serving. However, such augmentations pose significant challenges to inference efficiency under strict Service-Level Objectives (SLOs). Existing inference systems are agnostic to the dynamic execution behaviors induced by external calls and rely on fixed batch-level token budget, which leads to severe Head-of-Line (HoL) blocking and substantially reduced effective throughput. We present AugServe, an efficient augmented LLM inference serving framework that mitigates request queuing latency and improves effective throughput under external-call-augmented workloads. AugServe integrates state-aware request scheduling with dynamic batch-level token budgets to adapt to heterogeneous requests and their dynamically changing execution states. Experimental results show that AugServe achieves 6.5$\times$ and 4.7$\times$ higher effective throughput than vLLM and INFERCEPT, respectively.

Large language models are increasingly used in applications that call external tools, such as search engines or specialized models. These tools make AI systems more useful, but they also make serving many users efficiently more difficult. Tool-augmented requests may switch between model computation and waiting for external tool results. Existing serving systems often do not account for this switching behavior, so requests that are waiting for tools may delay other requests that are ready to use the GPU. We present AugServe, a system for serving tool-augmented language model applications more efficiently. Instead of processing requests mainly in arrival order, AugServe considers the current state of each request and decides which requests can make useful progress with the available GPU resources. It also dynamically decides how much work the GPU should handle at each step, based on the resources currently available. This research matters because tool-augmented LLMs are becoming the foundation of AI assistants and search agents. By addressing the new serving challenges introduced by external tool calls, AugServe helps these systems process requests more efficiently and better use limited GPU resources.