UltraHorizon: Benchmarking LLM-Agent Capabilities in Ultra Long-Horizon Scenarios

Autonomous agents have recently achieved remarkable progress across diverse domains, yet most evaluations focus on short-horizon, fully observable tasks. In contrast, many critical real-world tasks, such as large-scale software development, commercial investment, and scientific discovery, unfold in long-horizon and partially observable scenarios where success hinges on sustained reasoning, planning, memory management, and tool use. Existing benchmarks rarely capture these long-horizon challenges, leaving a gap in systematic evaluation. To bridge this gap, we introduce $\textbf{UltraHorizon}$, a novel benchmark that measures the foundational capabilities essential for complex real-world challenges. We use exploration as a unifying task across three distinct environments to validate these core competencies. Agents are designed in long-horizon discovery tasks where they must iteratively uncover hidden rules through sustained reasoning, planning, memory and tools management, and interaction with environments. Under the heaviest scale setting, trajectories average $\textbf{200k+}$ tokens and $\textbf{400+}$ tool calls, whereas in standard configurations they still exceed $\textbf{35k}$ tokens and involve more than $\textbf{60}$ tool calls on average. Our extensive experiments reveal that agents powered by state-of-the-art LLMs consistently underperform in these settings, whereas human participants achieve much higher scores, underscoring a persistent gap in agents' long-horizon exploration abilities. We also observe that simple scaling fails in our task. To better illustrate the failure of agents, we conduct an in-depth analysis of collected trajectories. We identify eight types of errors and attribute them to two primary causes: in-context locking and functional fundamental capability gaps.

While artificial intelligence has made remarkable progress on short, straightforward tasks, it still struggles with the messy, long-term challenges we face in the real world, such as scientific research or complex business investing. To test these limits, we created UltraHorizon, a massive new benchmark where AI agents must act like scientists to uncover hidden rules through continuous trial and error, often processing hundreds of thousands of words and making hundreds of decisions in a single attempt. Our experiments reveal that even the most advanced AI models fall far behind human participants in these scenarios. Furthermore, simply giving the AI more time or a larger interaction budget does not fix the problem. By analyzing their behavior, we found that AI agents typically fail because they get stubbornly stuck on a flaw we call "in-context locking", and because they lack the basic ability to reliably manage their memory and reasoning over long periods. Ultimately, UltraHorizon highlights a critical gap in current AI capabilities and provides a roadmap for building smarter, more persistent AI systems.