ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Ranking

Reinforcement learning (RL) has advanced LLM agents on verifiable tasks but remains challenging for open-ended tasks with vast solution spaces (e.g., complex travel planning). Lacking objective ground truth, current RL algorithms rely on reward models assigning scalar scores to individual responses. We contend such pointwise scoring induces discrimination collapse: reward model fails to distinguish subtle advantages among trajectories, compressing intra-group rewards into a narrow range. This drowns effective reward signals in reward model noise, causing optimization stagnation. To address this, we propose ArenaRL, a reinforcement learning paradigm shifting from pointwise scalar scoring to intra-group relative ranking. ArenaRL introduces a process-aware pairwise evaluation with multi-level rubrics for fine-grained relative scoring. Meanwhile, we construct an intra-group adversarial arena and devise a tournament-based ranking scheme to obtain stable advantage signals. ArenaRL achieves high-precision advantage estimation with only $O(N)$ computational complexity, striking a favourable balance between efficiency and accuracy. Furthermore, to address the lack of full-cycle benchmarks for open-ended agents, we introduce two high-quality benchmarks: Open-Travel and Open-DeepResearch, encompassing full training and multi-dimensional evaluation pipelines. Extensive experiments across three open-ended tasks validate the effectiveness of ArenaRL.

Large language model agents are increasingly used to solve complex open-ended tasks, such as planning a personalized trip or conducting deep research on the web. However, training these agents is difficult because there is often no single correct answer, and current reinforcement learning methods often rely on judge models to give each answer a scalar score. When many candidate answers are similarly good, these scores can be noisy and may hide the real differences between solutions. ArenaRL addresses this problem by replacing individual scalar scores with relative comparisons. Instead of asking how good each answer is on its own, ArenaRL lets candidate solutions compete in a tournament and learns from their relative rankings. The comparison also considers the agent's intermediate reasoning and tool use, not only the final answer. To support future research, we build two benchmarks: Open-Travel, which tests personalized travel planning, and Open-DeepResearch, which tests long-form web research. Our experiments on these benchmarks and open-ended writing tasks show that ArenaRL gives agents more reliable learning signals, helping them become better at complex real-world problem solving.