On Training Large Language Models for Long-Horizon Tasks: An Empirical Study of Horizon Length

Large language models (LLMs) have shown promise as interactive agents that solve tasks through extended sequences of environment interactions. While prior work has primarily focused on system-level optimizations or algorithmic improvements, the role of task horizon length in shaping training dynamics remains poorly understood. In this work, we present a systematic empirical study that examines horizon length through controlled task constructions. Specifically, we construct controlled tasks in which agents face identical decision rules and reasoning structures, but differ only in the length of action sequences required for successful completion. Our results reveal that increasing horizon length alone constitutes a training bottleneck, inducing severe training instability driven by exploration difficulties and credit assignment challenges. We demonstrate that horizon reduction is a key principle to address this limitation, stabilizing training and achieving better performance in long-horizon tasks. Moreover, we find that horizon reduction is related to stronger generalization across horizon lengths: models trained under reduced horizons generalize more effectively to longer-horizon variants at inference time, a phenomenon we refer to as horizon generalization.

Large language models are increasingly being used as AI agents, systems that solve tasks by taking many actions in sequence, like debugging code, browsing websites, or solving puzzles. But as these tasks grow longer, training these agents becomes unreliable, and we haven't fully understood why. To investigate this, we designed controlled experiments where the difficulty of reasoning stayed the same, but the number of steps required kept increasing. The results were clear: longer tasks alone can destabilize training. With more steps, the agent has more opportunities to make mistakes, and it becomes harder to figure out which earlier decisions actually led to success or failure. To fix this, we explored ways to reduce the number of effective steps the agent needs to take. Two simple strategies worked well: letting the agent use higher-level actions that bundle multiple steps together, and breaking the overall goal into subgoals. These changes led to more stable training and better performance on Sudoku, Rush Hour, and web navigation tasks. Perhaps most surprisingly, agents trained on shorter tasks were often able to handle longer, harder tasks at test time without any additional training. This points to a broader design principle: building tasks and action spaces that cut out unnecessary steps may be one of the most important factors in creating reliable, long-horizon AI agents.