CLIMB: Taming the LoRA Residency Cliff in Multi-LoRA Serving

Multi-tenant multi-LoRA serving multiplexes many LoRA adapters on a single GPU under high utilization, where most device memory is reserved for the KV cache, leaving only a small residency budget $K$ for adapters. In this regime, adapter readiness is atomic: if an adapter is not device-resident, the engine must perform a mandatory fetch, stalling shared execution and amplifying tail latency system-wide. With only $K$ residency slots, we identify a LoRA residency cliff: once the active adapter working set exceeds $K$, time-to-first-token (TTFT) tail latency can exhibit a congestion collapse rather than smooth degradation. To tame this cliff, we propose CLIMB, a minimal ingress controller that enforces feasibility-first admission by queueing non-resident adapters outside the engine, prioritizing critical (VIP) traffic, and rotating background adapters via round-robin. On a cliff-inducing workload, CLIMB averts collapse, reducing VIP TTFT p99 from 38.7 s to 13.1 s at matched throughput (10.66 rps) by keeping VIP engine latency near 0.13 s and shifting the residual tail into explicit ingress queueing. Overall, CLIMB shifts fetch-induced stalls from inside the engine to managed ingress queues, mitigating tail amplification without throughput loss in the evaluated settings.

Modern AI services often handle many customized tasks using the same large language model on one GPU. Each customization relies on a small add-on file that must be ready in GPU memory, but only a limited number of these files can fit at the same time. We found that when requests ask for too many different customizations at once, response times can suddenly become much worse rather than slowing down gradually. CLIMB prevents this by deciding which customizations are allowed into the serving system at each moment, so high-priority requests avoid expensive waiting inside the engine. In our experiments, this reduced severe slowdowns while keeping the service able to handle nearly the same number of requests.