Poster Mon, Jul 6, 2026 • 10:00 PM – 11:45 PM PDT HALL A #2105

NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents

Jingzhe Ding ⋅ Shengda Long ⋅ Changxin Pu ⋅ Ge Zhang ⋅ zhou huan ⋅ Hongwan Gao ⋅ Xiang Gao ⋅ Chao He ⋅ Yue Hou ⋅ FEI HU ⋅ Zhaojian Li ⋅ Weiran Shi ⋅ Zaiyuan Wang ⋅ Daoguang Zan ⋅ Chenchen Zhang ⋅ Xiaoxu Zhang ⋅ Chen Qizhi ⋅ cheng ⋅ Bo Deng ⋅ Qingshui Gu ⋅ Kai Hua ⋅ Juntao Lin ⋅ Pai Liu ⋅ Mingchen Li ⋅ Minghao Li ⋅ Xuanguang Pan ⋅ Zifan Peng ⋅ Yujia Qin ⋅ Yong Shan ⋅ Zhewen Tan ⋅ Haoran Wang ⋅ Tom Tang ⋅ Weihao Xie ⋅ Yishuo Yuan ⋅ Jiayu Zhang ⋅ Yunfei Zhao ⋅ He Zhu ⋅ LIYA ZHU ⋅ chenyangzou ⋅ Ming Ding ⋅ Jianpeng Jiao ⋅ Jiaheng Liu ⋅ Liam Liu ⋅ Qian Liu ⋅ Chongyang Tao ⋅ Jian Yang ⋅ Tong Yang ⋅ Zhaoxiang Zhang ⋅ Xinjie Chen ⋅ Wenhao Huang

Abstract

Recent advances in coding agents suggest rapid progress toward autonomous software development, yet existing benchmarks primarily evaluate short-horizon behaviors such as localized code generation, scaffolded completion, or repository repair, leaving it unclear whether agents can sustain coherent reasoning, planning, and execution over the extended horizons demanded by real-world repository construction. To address this gap, we introduce NL2Repo-Bench, a benchmark explicitly designed to evaluate the long-horizon repository generation from scratch: given only a single natural-language requirements document and an empty workspace, agents must autonomously design the architecture, manage dependencies, and produce a fully installable Python library. Experiments across state-of-the-art open- and closed-source models reveal that long-horizon repository generation remains largely unsolved, with even the strongest agents achieving merely 40\% average test pass rates and rarely completing an entire repository correctly. Further analysis identifies systematic long-horizon failure modes, including premature termination, loss of global coherence, fragile cross-file dependencies, and inadequate planning over hundreds of interaction steps. These results position NL2Repo-Bench as a rigorous, execution-based testbed for evaluating sustained agentic competence and highlight long-horizon reasoning as a key bottleneck for autonomous coding agents. Our data and code are available at https://github.com/multimodal-art-projection/NL2RepoBench.

Lay Summary

LLMs and Agents have become highly capable at solving short-horizon coding problems and generating small pieces of code. However, real software development is far more demanding: engineers must understand large codebases, coordinate changes across many files, manage dependencies, and carry out long sequences of decisions over time. We introduce NL2Repo-Bench, a benchmark designed to evaluate whether agents can handle this kind of real-world software creation. Instead of modifying a few existing functions, the agents receive only a natural-language requirements document and must build working functionality for an entire repository starting from an empty workspace. To succeed, the agent must make many connected decisions over a long development process, similar to how human developers gradually build software projects step by step. Our results show that this remains extremely difficult for current models and agents. Even the strongest coding agents often fail when tasks require maintaining consistency across many files, tracking long-term goals, and recovering from earlier mistakes during extended development sessions. We hope NL2Repo-Bench can help researchers build AI coding assistants that are better suited for real-world software engineering tasks.