CATArena: Evaluating Evolutionary Capabilities of Code Agents via Iterative Tournaments

Current evaluation for Large Language Model (LLM) code agents predominantly focus on generating functional code in single-turn scenarios, which fails to evaluate the agent's capability for continuous code optimization and multi-turn iterative development. To bridge this gap, we introduce CATArena, a framework designed to evaluate the evolutionary capabilities of code agents via iterative tournaments. Agents engage in multi-turn tournaments and continuously refine their code through self-reflection and peer-learning based on comprehensive execution feedback. For evaluation, we propose a dual-metric system to decouple static generation proficiency from evolutionary potential. Extensive experiments reveal that an agent's evolutionary potential is not strictly correlated with its initial proficiency. Our analysis further reveals that current agents struggle to concurrently leverage both peer-learning and self-reflection for effective performance gains. Furthermore, the results validate CATArena's high extensibility and resistance to variance tasks, establishing it as a continuous and reliable standard for assessing the evolutionary capability of LLM code agents.

LLM Code Agents are often tested on whether they can produce a correct solution in a single attempt. But in real software development, improvement usually happens over multiple rounds: code is tested, revised, compared with other solutions, and refined again. Our work introduces CATArena, a benchmark designed to measure this more realistic ability to improve over time. In CATArena, code agents compete in repeated rounds and update their programs based on feedback from execution results, their own past mistakes, and ideas learned from other agents. We also introduce two separate evaluation measures so that we can distinguish between an agent that starts strong and an agent that learns and improves effectively. Our experiments show that these are not the same skill: a system that writes better code at the beginning does not necessarily improve more over time. We also find that today’s code agents still have difficulty combining self-correction with learning from others. Overall, CATArena offers a practical and reliable way to evaluate how well AI coding systems can adapt, improve, and evolve through iteration.