LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents

Role specialization in multi-LLM agent systems is often realized via multi-LoRA, where agents share a pretrained backbone and differ only through lightweight adapters. Despite sharing base model weights, each agent independently builds and stores its own KV cache for the same long, tool-augmented trajectories, incurring substantial memory and compute overhead. Existing KV cache sharing methods largely overlook this multi-LoRA setting. We observe that, across agents, cache differences are dominated by adapter outputs, while activations from the shared pretrained backbone remain highly similar. Based on this observation, we propose LRAgent, a KV cache sharing framework for multi-LoRA agents that decomposes the cache into a shared base component from the pretrained weights and an adapter-dependent component from LoRA weights. LRAgent reduces memory overhead by sharing the base component and storing the adapter component in its inherent low-rank form, and further reduces compute overhead, enabled by shared-$A$ multi-LoRA architectures, by also sharing the low-rank cache and avoiding redundant computations for contexts already processed by other agents. To efficiently reconstruct adapter contributions at runtime, we introduce Flash-LoRA-Attention, a kernel that reorders attention computation to avoid materializing the low-rank cache to full dimension. LRAgent achieves throughput and time-to-first-token latency close to fully shared caching, while preserving accuracy near the non-shared caching baseline across agentic question-answering benchmarks.

Modern AI systems often use multiple LLM agents that work together to solve complex tasks. For example, one agent plans, another uses external tools like web search, and a third reviews the answer. To specialize each agent cheaply, developers add small LoRA adapter modules on top of one shared base model. However, each agent still keeps its own memory of the conversation so far, even though most of that memory comes from the shared base model and should be nearly identical across agents. This wastes a lot of GPU memory and forces the same text to be reprocessed multiple times, making the system slow. We noticed that the memory each agent builds can be split into two parts: a large shared part from the base model, and a tiny adapter-specific part. Our method, LRAgent, stores the shared part only once and keeps the adapter-specific part in a highly compressed form. We also designed a custom GPU kernel that uses this compressed form directly, avoiding wasteful expansion. This makes multi-agent AI systems much faster and lighter on memory without sacrificing answer quality, enabling longer, more capable agent interactions on the same hardware.