SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

We present SWE-Bench Pro, a comprehensive benchmark designed to evaluate software engineering capabilities through complex, realistic programming challenges. This benchmark extends beyond traditional algorithmic problems to encompass the full spectrum of professional software development tasks. The dataset comprises 1,865 problems sourced from 41 active software engineering repositories, spanning 123 unique programming languages and various application domains. The benchmark is structured into public and private components, with public access to problems from 11 repositories and private evaluation sets from 12 repositories across 4 distinct problem categories. SWE-Bench Pro addresses limitations of existing evaluation frameworks by incorporating problems that reflect real-world software engineering scenarios, including substantial codebases, complex enterprise applications, and multi-file projects requiring sophisticated reasoning and code modification skills. Problems range from early-stage startup environments to enterprise-level applications, with the private commercial set remaining inaccessible to maintain evaluation integrity while enabling public access to representative problems for professional development. Our evaluation methodology employs diverse coding approaches and models under controlled conditions, ensuring robust performance assessment across multiple programming paradigms. Results demonstrate significant performance variations across different problem categories, with traditional algorithmic challenges showing notably higher success rates compared to complex, multi-file engineering tasks. The benchmark reveals substantial gaps in current capabilities for handling real-world software engineering scenarios, particularly in areas requiring deep contextual understanding, cross-file reasoning, and integration with existing large-scale systems. This work contributes a more comprehensive and realistic evaluation framework for assessing software engineering capabilities, providing insights into current limitations and establishing a foundation for future development in automated software engineering tools and methodologies.

AI coding assistants have become a daily tool for software developers, and the field tracks their progress with benchmarks: standardized test sets of programming problems. The most widely used benchmark, SWE-Bench, draws problems from popular open-source projects on GitHub. Two issues have grown harder to ignore. First, those same projects sit in the training data of the models being tested, so high scores can reflect memorization rather than real problem solving. Second, the problems are short. Many require only one or two lines of code to fix, which is not what a working software engineer faces day to day. We built SWE-Bench Pro to address both problems. It contains 1,865 problems drawn from 41 actively maintained codebases, including 18 proprietary repositories we acquired from startups under formal partnership agreements. The other repositories use copyleft licenses (such as GPL) that legally restrict their use in commercial model training. The problems are longer and span multiple files: 107 lines of code and 4 files on average, versus 12 lines and a single file on typical existing benchmarks. Every problem is reviewed by a professional software engineer who clarifies the task description, removes ambiguous tests, and confirms the problem is solvable. When we evaluate today's leading AI coding agents on SWE-Bench Pro, the best models solve fewer than 45% of the public problems and fewer than 20% of the proprietary ones, compared to over 70% on the older SWE-Bench Verified benchmark. We also categorize the ways models fail, from misunderstanding the task to producing code that does not compile. SWE-Bench Pro gives the community a harder, more realistic, and harder-to-game yardstick for measuring progress toward AI agents that can do professional software engineering.