EvoClaw: Evaluating AI Agents on Continuous Software Evolution

Real-world software must continuously evolve to meet ever-changing and open-ended requirements. AI agents, increasingly deployed as long-running systems, are now entrusted to drive this evolution. Yet, existing benchmarks evaluate agents on isolated, one-off coding tasks, neglecting the temporal dependencies and technical debt inherent in real-world software evolution. To bridge this gap, we introduce DeepCommit, an agentic pipeline that reconstructs verifiable Milestone DAGs from noisy commit logs, where milestones are defined as functionally cohesive development goals. These executable sequences enable SWE-Milestone, a benchmark that evaluates agents on streams of milestone-level tasks, requiring them to sustain system integrity and limit error accumulation, dimensions of long-term software evolution largely missing from current benchmarks. Our evaluation of 12 frontier models across 4 agent frameworks reveals a critical vulnerability: overall performance scores drop significantly from >80% on isolated tasks to at most 38% in continuous settings, exposing agents' profound struggle with long-term maintenance and error propagation. Project website: swe-milestone.com

Today's AI coding agents are usually tested on isolated tasks, such as fixing a single bug or adding a feature. But real software development is a marathon where changes accumulate, and early mistakes snowball into major failures. How well do top AI agents handle this continuous reality? To find out, we built DeepCommit to turn messy git histories into ordered milestones, and EvoClaw, a benchmark of 98 milestones across seven projects. We tested 12 frontier models using four agent frameworks, including Claude Code and OpenHands. The results were stark. Agents solved over 80% of tasks in isolation, but their success plummeted to under 38% during continuous evolution, with only 13% of projects fully completed. While they can add new features, they constantly break existing code. Unresolved bugs poison subsequent tasks until the codebase degrades and the models give up. Until this gap is bridged, AI agents cannot autonomously maintain real software.