ExpWeaver: LLM Agents Learn from Experience via Latent RAG

Experience learning has achieved promising results in enhancing LLM agent planning and reasoning by integrating past interactions as reusable knowledge. However, existing methods remain confined to explicit text space—retrieving experiences via semantic similarity and concatenating them into the context window, leading to substantial token overhead and a decoupled architecture that separates retrieval from generation. To address these limitations, we propose ExpWeaver, a framework that enables LLM agents to learn from experience via latent retrieval-augmented generation, without requiring a separate RAG module. ExpWeaver encodes experiences using the LLM’s own hidden states, retrieves relevant experiences directly in latent space at each decoding step, and integrates them through cross-attention aggregation and gated residual mechanisms. The entire pipeline is optimized end-to-end with reinforcement learning, supporting both generative and ranking tasks. We evaluate ExpWeaver on 13 diverse tasks spanning question answering, reasoning, coding, scientific prediction, and recommendation. Results demonstrate that: (1) ExpWeaver achieves state-of-the-art on 12 out of 13 tasks, outperforming the strongest baseline by over 6.8%; (2) ExpWeaver maintains token efficiency comparable to non-retrieval baselines while text-based retrieval methods require 1.5–2× more tokens; and (3) ExpWeaver exhibits superior cross-domain generalization, outperforming the strongest baseline by 16.32% under zero-shot transfer and 15.21% under few-shot transfer. Our code for ExpWeaver is released at https://github.com/ulab-uiuc/ExpWeaver.

Large language model agents can solve many tasks, but they often struggle to make efficient use of what they have learned from past attempts. Existing methods usually store past experiences as text and add them to the model’s input, which increases cost and is limited by the model’s context length. This paper introduces ExpWeaver, a framework that helps language model agents learn from experience in a more efficient way. Instead of adding long textual memories, ExpWeaver stores past experiences in the model’s internal representation space and lets the model retrieve and use them while solving new problems. This removes the need for a separate retrieval system and reduces the number of extra tokens needed. We evaluate ExpWeaver on 13 tasks, including question answering, reasoning, coding, scientific prediction, and recommendation. ExpWeaver achieves the best performance on 12 of the 13 tasks, uses far fewer tokens than text-based retrieval methods, and transfers better to new domains. These results show that learning from experience inside the model’s own representation space can make language model agents more accurate, efficient, and adaptable.