SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?

Code performance optimization is paramount in real-world software engineering and critical for production-level systems. While Large Language Models (LLMs) have demonstrated impressive capabilities in code generation and bug fixing, their proficiency in enhancing code performance at the repository level remains largely unexplored. To address this gap, we introduce SWE-Perf, the first benchmark specifically designed to systematically evaluate LLMs on code performance optimization tasks within authentic repository contexts. SWE-Perf comprises 140 carefully curated instances, each derived from performance-improving pull requests from popular GitHub repositories. Each benchmark instance includes the relevant codebase, target functions, performance-related tests, expert-authored patches, and executable environments. Through a comprehensive evaluation of representative methods that span file-level and repo-level approaches (e.g., Agentless and OpenHands), we reveal a substantial capability gap between existing LLMs and expert-level optimization performance, highlighting critical research opportunities in this emerging field.

Software teams often need to make programs run faster or use fewer resources, but this is hard because real projects are large and changes must still keep the software correct. Although recent AI coding assistants are good at writing code and fixing some bugs, it is still unclear how well they can improve the speed of code in realistic projects. This paper introduces SWE-Perf, a new test collection for measuring how well AI systems handle real-world code performance improvement tasks. SWE-Perf contains 140 tasks taken from actual improvements made in popular open-source projects. Each task includes the relevant project code, the part of the code that should be improved, tests that check performance, expert-written solutions, and an environment where the task can be run. By testing several representative AI coding systems, we find that current systems are still far from matching expert developers on these tasks. This suggests that improving software performance remains an important and challenging direction for future AI coding research.