Poster

SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models

Xu jingxuan ⋅ Ken Deng ⋅ Weihao Li ⋅ Songwei Yu ⋅ Haoyang Huang ⋅ Yifan Yao ⋅ Huaixi Tang ⋅ Zhiyi Lai ⋅ Kepeng Lei ⋅ Zizheng Zhan ⋅ Yanan Wu ⋅ Chenchen Zhang ⋅ Xinping Lei ⋅ Wenqiang Zhu ⋅ wen xiang ⋅ Zongxian Feng ⋅ Han Li ⋅ Junqi Xiong ⋅ Dailin Li ⋅ GAO Zuchen ⋅ Kun Wu ⋅ Yuanxing Zhang ⋅ Wuxuan Gong ⋅ Ziyuan Gao ⋅ Guanxiang Wang ⋅ Yirong Xue ⋅ Mengfei Xie ⋅ Xiaojiang Zhang ⋅ Jinghui Wang ⋅ Wenhao Zhuang ⋅ Zheng Lin ⋅ Huiming Wang ⋅ Zhaoxiang Zhang ⋅ Yuqun Zhang ⋅ Haotian Zhang ⋅ Ming Sun ⋅ chenbin ⋅ Jiaheng Liu

Abstract

Evaluating large language models (LLMs) for software engineering has been limited by narrow task coverage, language bias, and insufficient alignment with real-world developer workflows. Existing benchmarks often focus on algorithmic problems or Python-centric bug fixing, leaving critical dimensions of software engineering underexplored. To address these gaps, we introduce SWE-Compass, a comprehensive benchmark that unifies heterogeneous code-related evaluations into a structured and production-aligned framework. SWE-Compass spans 8 task types, 8 programming scenarios, and 10 programming languages, with 2000 high-quality instances curated from authentic GitHub pull requests and refined through systematic filtering and validation. We benchmark ten state-of-the-art LLMs under two agentic frameworks, SWE-Agent and Claude Code, revealing a clear hierarchy of difficulty across task types, languages, and scenarios. Moreover, by aligning evaluation with real-world developer practices, we hope SWE-Compass can provide a rigorous and reproducible foundation for diagnosing and advancing agentic coding capabilities in large language models.