Experience-Evolving Multi-Turn Tool-Use Agent with Hybrid Episodic–Procedural Memory

As intents unfold and environments change, multi-turn agents face continuously shifting decision contexts. Although reusing past experience is intuitively appealing, existing approaches remain limited: full trajectories are often too context-specific to transfer, while tool-level reuse ignores the context and environment. In this paper, we introduce a hybrid episodic–procedural memory strategy (H-EPM) that enables experience-evolution of multi-turn tool-use policies, by adaptively reusing partially overlapping successful experiences in both inference and training. Inspired by human episodic–procedural integration, we build a tool graph from accumulated trajectories, where recurring tool-to-tool dependencies capture procedural routines and each edge is augmented with a compact episodic summaries of relevant context. At inference, the agent dynamically balances episodic recall for contextual reasoning and procedural execution for routine steps. Beyond inference, H-EPM introduces a memory-guided reinforcement learning paradigm that directly addresses a core challenge in multi-turn agent RL: ineffective exploration over long trajectories. By biasing exploration toward historically successful tool transitions, H-EPM learns a stronger policy that generalizes during inference without relying on domain-specific experience collection. Experiments show that H-EPM consistently delivers substantial inference-time gains over strong baselines across multi-turn tool-use benchmarks, reaching up to 50%+. It also boosts RL policy performance, achieving up to 40%+ improvement on out-of-distribution tasks. Our code is available at https://github.com/LISijia-dev/H-EPM.

As AI agents solve tasks through long conversations and tool use, they must adapt to constantly changing contexts. Reusing past experience seems useful, but existing methods either rely on entire past trajectories that are too situation-specific or reuse isolated tools without understanding the surrounding context. In this work, we introduce Hybrid Episodic–Procedural Memory (H-EPM), a memory framework inspired by how humans combine specific experiences with learned routines. H-EPM helps agents reuse both contextual experiences and recurring tool-use patterns, allowing them to better adapt to new situations. We further show that memory can improve not only inference, but also reinforcement learning itself. By guiding exploration toward historically successful behaviors, H-EPM helps agents learn more effectively in multi-turn tasks. Experiments on multiple multi-turn tool-use benchmarks show that H-EPM significantly improves both agent performance and generalization, achieving large gains during inference and reinforcement learning.