ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning

Agentic reinforcement learning (ARL) has rapidly gained attention as a promising paradigm for training agents to solve complex, multi-step interactive tasks. Despite encouraging early results, ARL remains highly unstable, often leading to training collapse. This instability limits scalability to larger environments and longer interaction horizons, and constrains systematic exploration of algorithmic design choices. In this paper, we first propose ARLArena, a stable training recipe and systematic analysis framework that examines training stability in a controlled and reproducible setting. ARLArena first constructs a clean and standardized testbed. Then, we decompose policy gradient into four core design dimensions and assess the performance and stability of each dimension. Through this fine-grained analysis, we propose SAMPO, a stable agentic policy optimization method designed to mitigate the dominant sources of instability in ARL. Empirically, SAMPO achieves consistently stable training and strong performance across diverse agentic tasks. Our code is open-sourced at: https://github.com/WillDreamer/ARL-Arena.git

AI assistants are increasingly asked to perform long sequences of actions on our behalf — browsing websites, navigating virtual environments, or solving multi-step problems with external tools. Teaching them through trial and error, a method called reinforcement learning, frequently goes badly wrong: training suddenly collapses, the assistant forgets what it learned, and progress unravels. This fragility has blocked scientific progress on more ambitious, longer-horizon agent tasks. We built a controlled "training arena" to systematically diagnose which design choices actually matter for stability. By decomposing the learning algorithm into four interchangeable parts and stress-testing each one in isolation, we traced the dominant cause of collapse to a small set of misjudged decisions whose influence is amplified during training until everything else destabilizes. Guided by these findings, we designed SAMPO, a new training recipe that combines the most stable choice along each dimension into a single coherent method. SAMPO trains AI agents reliably across tasks ranging from simulated household chores to online shopping to puzzle-solving, improving success rates by 25% on average over the standard approach. A small open model trained with SAMPO even outperforms much larger commercial systems like GPT-5.2 on these agent tasks, offering a reproducible foundation for scaling AI agents to longer, more complex behaviors.