CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control

Batch inference for agentic workloads stresses the GPU key–value (KV) cache in a sustained and cumulative manner, often causing severe throughput degradation well before memory capacity is exhausted. We identify this phenomenon as middle-phase thrashing, a previously under-characterized pathology in which cache efficiency collapses as long-lived agents accumulate state over time. We argue that mitigating this pathology requires moving beyond reactive, request-level cache management to proactive, agent-level admission control. Drawing inspiration from congestion control in distributed systems, we view the KV cache as a shared resource whose efficient utilization depends on feedback-driven regulation. Based on this insight, we present CONCUR, a lightweight control layer that regulates agent admission to bound aggregate cache pressure while preserving execution continuity. CONCUR adapts a cache-aware control algorithm to dynamically adjust the number of active agents using runtime cache signals. Across large models and real-world agent workloads, CONCUR prevents middle-phase thrashing and improves batch inference throughput by up to 4.09× on Qwen3-32B and 1.90× on DeepSeek-V3, while remaining compatible with existing LLM serving systems.

Modern AI assistants often need to handle many long-running tasks at the same time, such as answering questions, planning actions, or interacting with tools. As these tasks continue, they gradually consume more and more GPU memory, which can suddenly slow the entire system down even when plenty of memory still appears to be available. We identify this hidden slowdown problem, which we call “middle-phase thrashing,” and show that it is a major bottleneck for large-scale AI services. To address this issue, we developed CONCUR, a lightweight system that carefully controls how many AI agents are active at once. Inspired by how internet traffic is managed to avoid congestion, CONCUR continuously monitors GPU memory behavior and adjusts workload levels before severe slowdowns occur. Across a range of popular AI models and realistic workloads, CONCUR significantly improves system efficiency and keeps performance stable over long periods of use. Our method increases processing throughput by up to 4× while remaining easy to integrate into existing AI serving systems.