Poster

SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models

Jingxuan Xu ⋅ Ken Deng ⋅ Weihao Li ⋅ Songwei Yu ⋅ Haoyang Huang ⋅ Xinping Lei ⋅ Yifan Yao ⋅ Huaixi Tang ⋅ Zhiyi Lai ⋅ Kepeng Lei ⋅ Zizheng Zhan ⋅ Yanan Wu ⋅ Chenchen Zhang ⋅ Wenqiang Zhu ⋅ wen xiang ⋅ Zongxian Feng ⋅ Han Li ⋅ Junqi Xiong ⋅ Dailin Li ⋅ GAO Zuchen ⋅ Kun Wu ⋅ Yuanxing Zhang ⋅ Wuxuan Gong ⋅ Ziyuan Gao ⋅ Guanxiang Wang ⋅ Yirong Xue ⋅ Mengfei Xie ⋅ Xiaojiang Zhang ⋅ Jinghui Wang ⋅ Wenhao Zhuang ⋅ Zheng Lin ⋅ Huiming Wang ⋅ Zhaoxiang Zhang ⋅ Yuqun Zhang ⋅ Haotian Zhang ⋅ Ming Sun ⋅ chenbin ⋅ Jiaheng Liu

Project Page

Abstract

Evaluating large language models (LLMs) for software engineering has been limited by narrow task coverage, language bias, and insufficient alignment with real-world developer workflows. Existing benchmarks often focus on algorithmic problems or Python-centric bug fixing, leaving critical dimensions of software engineering underexplored. To address these gaps, we introduce SWE-Compass, a comprehensive benchmark that unifies heterogeneous code-related evaluations into a structured and production-aligned framework. SWE-Compass spans 8 task types, 8 programming scenarios, and 10 programming languages, with 2000 high-quality instances curated from authentic GitHub pull requests and refined through systematic filtering and validation. We benchmark ten state-of-the-art LLMs under two agentic frameworks, SWE-Agent and Claude Code, revealing a clear hierarchy of difficulty across task types, languages, and scenarios. Moreover, by aligning evaluation with real-world developer practices, we hope SWE-Compass can provide a rigorous and reproducible foundation for diagnosing and advancing agentic coding capabilities in large language models.

Lay Summary

AI coding assistants are becoming increasingly capable, but it is still hard to know whether they can handle the kinds of software engineering work that developers face in real projects. Many existing evaluations focus on small programming exercises or narrow bug-fixing tasks, so they miss important activities such as adding features, improving code, writing tests, understanding large codebases, and handling deployment. We introduce SWE-Compass, a benchmark designed to test AI coding agents across a broader and more realistic range of software engineering tasks. It contains 2,000 carefully verified tasks collected from real GitHub pull requests, covering 8 task types, 8 development scenarios, and 10 programming languages. Each task is paired with an executable environment so that model outputs can be checked in a more reliable and reproducible way. Using SWE-Compass, we evaluate leading language models under two coding-agent workflows. Our results show that today’s systems still struggle with many realistic tasks, especially when they need to understand requirements, reason across code, and use execution feedback effectively. We hope this benchmark will help researchers and developers build more reliable AI coding assistants.