CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding

LLM-powered coding agents are reshaping the development paradigm. However, existing evaluation systems, neither traditional tests for humans nor benchmarks for LLMs, fail to capture this shift, excluding problems that require both human reasoning to guide solutions and AI efficiency for implementation. We introduce CentaurEval, a unified, ecologically valid benchmark for measuring human-in-the-loop value in coding. CentaurEval's core innovation is its "Collaboration-Necessary" problem templates, which are intractable for standalone LLMs or humans, but solvable through effective collaboration. CentaurEval dynamically instantiates tasks from 45 templates, providing a standardized IDE for humans and a reproducible 450-task toolkit for LLMs. We benchmark 45 participants against 5 LLMs under 4 levels of human intervention. Results show that while LLMs or humans alone achieve poor pass rates (0.67% and 18.89%), human–AI collaboration significantly improves to 31.11%. Our analysis reveals an emerging co-reasoning partnership, challenging the traditional human-tool hierarchy by showing that strategic breakthroughs can originate from either humans or AI.

Many software developers now work with AI coding assistants, but it is still hard to measure what humans uniquely contribute in this new workflow. Traditional coding tests usually ask people or AI systems to solve clearly defined programming problems, while real work often starts with messy requirements, missing information, and decisions about what problem should be solved in the first place. We built CentaurEval, a benchmark designed to evaluate coding tasks where neither a human nor an AI assistant is expected to perform well alone. In these tasks, humans can guide strategy, interpret unclear requirements, and check whether the solution makes sense, while AI assistants can help implement and iterate quickly. We tested 45 experienced users and 5 modern AI coding systems under different levels of human involvement. The results show that human-AI teams solved substantially more tasks than either humans or AI systems working alone. This suggests that the value of developers in the AI era is not just writing code, but steering, questioning, and validating AI-generated solutions. CentaurEval can help researchers track how this human-AI boundary changes as coding agents improve.