Comparing Developer and LLM Biases in Code Evaluation

LLMs are increasingly used as judges in code applications, yet their efficacy remains untested in realistic code settings that often feature partial context and ambiguous intent. We present TRACE (Tool for Rubric Analysis in Code Evaluation), a framework that evaluates LLM judges’ ability to predict human preferences and automatically extracts rubric items to highlight systematic differences in how humans and models weight these criteria. Across three modalities---IDE autocompletion, chat-based programming, and instructed code editing---we benchmark 13 judge models spanning general-purpose LLMs, specialized judge models, and reward models. Among these models, even the best judges align with human preference only 6--18% above random chance. TRACE identifies 8 significantly misaligned rubric items across interaction modalities. For example, in chat-based coding, judges prefer longer code explanations while humans prefer shorter ones. Many of these gaps occur within existing code quality dimensions, showing that current LLM judges remain misaligned with human evaluation in realistic coding workflows.