ExpWeaver: LLM Agents Learn from Experience via Latent RAG

Experience learning has achieved promising results in enhancing LLM agent planning and reasoning by integrating past interactions as reusable knowledge. However, existing methods remain confined to explicit text space---retrieving experiences via semantic similarity and concatenating them into the context window, leading to substantial token overhead and a decoupled architecture that separates retrieval from generation. To address these limitations, we propose \method, a framework that enables LLM agents to learn from experience via latent retrieval-augmented generation, without requiring a separate RAG module. \method encodes experiences using the LLM's own hidden states, retrieves relevant experiences directly in latent space at each decoding step, and integrates them through cross-attention aggregation and gated residual mechanisms. The entire pipeline is optimized end-to-end with reinforcement learning, supporting both generative and ranking tasks. We evaluate \method on 13 diverse tasks spanning question answering, reasoning, coding, scientific prediction, and recommendation. Results demonstrate that: (1) \method achieves state-of-the-art on 12 out of 13 tasks, outperforming the strongest baseline by over 6.8\%; (2) \method maintains token efficiency comparable to non-retrieval baselines while text-based retrieval methods require 1.5--2$\times$ more tokens; and (3) \method exhibits superior cross-domain generalization, outperforming the strongest baseline by 16.32\% under zero-shot transfer and 15.21\% under few-shot transfer.