From Interactions to Principles: Experience-Driven Self-Distillation for Evolving LLM Agents

LLM agents have achieved strong performance in tool-augmented reasoning, but most remain largely stateless: after each episode, the agent discards interaction traces and does not accumulate reusable strategies. Prior work either stores raw trajectories for case-based reuse or relies on external teacher models to write reflections, which limits generalization or leaves the agent’s policy unchanged. We introduce EvolveR, an experience-driven framework that allows an agent to improve using its own interaction history. EvolveR maintains an experience base of distilled strategic principles derived from past trajectories. In an offline phase, the agent self-distills successful and failed trajectories into concise principles, applies semantic deduplication, and assigns each principle an empirical utility score for maintenance and pruning. In an online phase, the agent retrieves top-ranked principles to guide reasoning and tool usage, generating new trajectories. We then perform policy evolution with reinforcement learning on these experience-conditioned trajectories, reinforcing behaviors that effectively retrieve and apply useful principles. We demonstrate the effectiveness of EvolveR on complex multi-hop question-answering benchmarks, where it achieves superior performance over strong agentic baselines. Our work presents a comprehensive blueprint for agents that learn not only from external data but also from the consequences of their own actions, paving the way for more autonomous and continuously improving systems.

Today's AI assistants can use tools like search engines to answer questions, but unlike humans, they cannot learn from their own successes and failures. If they make a mistake on one problem, they will repeat the same mistake on similar problems later. Current approaches try to fill these gaps by fetching information from external sources, but they don't help the assistant become a better problem-solver over time. We introduce EvolveR, a framework that gives AI assistants the ability to improve through experience. After solving (or failing to solve) a task, the system reviews what happened and extracts general, reusable strategies—similar to how a student might write down lessons learned after taking an exam. These strategies are stored and later retrieved to guide the assistant when it faces new, unfamiliar problems. As this cycle repeats, the assistant's performance continuously improves through its own experience, without needing external retraining. On challenging multi-step question-answering tasks that require piecing together information from multiple sources, EvolveR consistently outperformed strong existing methods. More broadly, this work offers a blueprint for building AI systems that genuinely grow more capable through their own actions—a step toward more autonomous and continuously improving agents.