Beyond Pass@k: Developing Human-Centric Evaluation Metrics for AI Pair Programmers

Current benchmarks for AI code generation mea- sure functional correctness through metrics such as pass@k, but ignore the dimensions of code quality that determine whether generated code is useful to the developer who must read, adapt, and maintain it. We introduce the Human-Centric Composite Score (HCCS), a composite evalua- tion metric comprising three sub-scores measur- ing Readability, Style Adherence, and Review Ease, and construct PairEval, a 1,800-item bench- mark of realistic coding tasks paired with devel- oper style profiles and human review annotations (n = 120 developers). Evaluation of six model families reveals a strong but imperfect correlation between pass@1 and HCCS (r = 0.92, p < 0.05, n = 6 models), confirming that correctness and human-centricity are related but structurally dis- tinct: Claude 3.5 Sonnet achieves the highest HCCS among base models despite not achieving the highest pass@1. A human study shows that HCCS-optimised code reduces mean review time by 28% and raises defect detection rates by 15 percentage points. Fine-tuning with HCCS pref- erence pairs raises HCCS by 16 points absolute at a correctness tax of only ∆pass@1 = −0.010.