LOCA-bench: Benchmarking Language Agents Under Controllable and Extreme Context Growth

Frontier large language models (LLMs) are increasingly capable of carrying out long-running, real-world tasks. However, as the amount of context grows, their reliability often deteriorate, a phenomenon known as "context rot". Existing long-context benchmarks primarily focus on single-step settings that evaluate a model’s ability to retrieve information from a long snippet. In realistic scenarios, however, LLMs often need to act as agents that explore environments, follow instructions and plans, extract useful information, and predict correct actions under a dynamically growing context. To assess language agents in such settings, we introduce LOCA-bench (a benchmark for LOng-Context Agents). Given a task prompt, LOCA-bench leverages automated and scalable control of environment states to regulate the agent’s context length. This design enables LOCA-bench to extend the context length potentially to infinity in a controlled way while keeping the underlying task semantics fixed. LOCA-bench evaluates language agents as a combination of models and scaffolds, including various context management strategies. While agent performance generally degrades as the environment states grow more complex, advanced context management techniques can substantially improve the overall success rate. We will open-source LOCA-bench to provide a platform for evaluating models and scaffolds in long-context, agentic scenarios.

Large language models are getting better at doing complex tasks that take many steps, such as searching through files, following instructions, using tools, and making decisions. But when these tasks become long and the model has to keep track of more and more information, its performance often gets worse. This problem is called “context rot”: the model’s useful understanding starts to decay as the context becomes too large or messy. Most existing benchmarks only test whether a model can find a piece of information inside a long document. But real-world agents face a harder problem: they must work through changing environments, gather information over time, manage growing context, and still choose the right actions. To study this, the authors introduce LOCA-bench, a benchmark for testing long-context language agents. LOCA-bench can control how much environmental information is shown to an agent while keeping the actual task the same. This lets researchers test what happens as the context grows larger and more complicated. The benchmark evaluates not just the language model itself, but the full agent system, including memory, planning, and context-management methods. The results show that agents usually perform worse as the environment becomes more complex, but better context-management strategies can significantly improve success rates. In simple terms: LOCA-bench is a new testbed for measuring how well AI agents can stay reliable when tasks become long, messy, and information-heavy.