Paper #51: JudgeBreak: Is Taste Hidden in the Layers? Stress-Testing Reward Models and LLM Judges with Representation-Based Subjective Evaluation

Reward models and LLM judges are the default for text evaluation, yet linear probes on LLM hidden states, a far simpler alternative remain overlooked. We introduce JudgeBreak, a benchmark of 95 evaluation settings spanning creative writing, summarization, safety, 10 languages, and 33 writing genres. Hidden-state probes with cross-validated layer selection on Qwen3-32B achieve 78.1% mean pairwise accuracy versus 70.1% for the best reward model or zero-shot GPT-5.4 judge, winning 43 of 95 settings. A DSPy-optimized GPT-5.4 judge narrows the gap on some tasks but remains order-sensitive and still trails probes on the hardest creative and specialized domains. Gains are largest where external judges fail most: on expert-annotated creative writing, probes reach 96% per-dimension accuracy while reward models are systematically inverted at 11–15%, preferring lower- quality text across all 14 dimensions, an 80-point gap suggesting RLHF-trained signals can anti-correlate with expert aesthetic judgment. On specialized domains like academic review and advertising copy, probes similarly dominate (85% vs. 57%). Two factors explain these results. First, domain proximity: reward models encode a narrow quality signal shaped by their training distribution, whereas probes recover broader concepts from internal representations. Layer selection is crucial, improving accuracy by 8 points on average and generalizing across six models from four families (7B–32B), though selected layers do not transfer across architectures. Second, label coherence: when annotators disagree about quality, all methods plateau near chance, cross-cultural data from PRISM and creativity criteria from Chatbot Arena both cap at 57%. Together, these findings show that text quality is often more recoverable from internal representations than from external evaluators, especially for subjective and domain-specialized tasks.