RubricRobustness: Evaluating the Sensitivity of Rubrics-Based Benchmarks to Simple Perturbations

The advancement of Large Language Models (LLMs) into higher-level reasoning domains has rendered traditional heuristic evaluators insufficient for long-form open-ended responses, precipitating the widespread adoption of rubric-based benchmarks. While these frameworks utilize expert-curated criteria and LLM-as-a-judge to assess open-ended generation, the intrinsic robustness of these evaluation harnesses to fundamental validity assessments remains critically under-investigated. To bridge this gap, we introduce RubricRobustness, a systematic sensitivity analysis framework that subjects these benchmarks to three common sense perturbations: *semantic negation*, *stochastic deletion* and *irrelevant addition*. We investigate the extent to which manipulating the semantic veracity of a model's response impacts its resulting score by applying the robustness framework to two of the most popular rubrics-based benchmarks: HealthBench and WildBench. Our findings reveal systematic vulnerabilities: while both benchmarks respond sharply to semantic negation (e.g., degradation slopes of approximately $-0.38$ on HealthBench and $-0.55$ on WildBench), they are substantially less responsive to irrelevant addition, often requiring over 35% of sentences to be perturbed before inducing even a 25% score drop. We argue that perturbation-based sensitivity analyses of this form are a necessary prerequisite for validating rubric coverage, ensuring that automated evaluation frameworks reliably penalize basic semantic failures. We will release our framework as an open-source tool for building more resilient benchmarks.

As AI language models take on higher-stakes work, such as giving medical advice or answering open-ended questions, we increasingly rely on other AI systems to grade their answers. These automatic graders score a response by checking it against a long list of expert-written criteria. This raises a basic question: can such a grader actually tell when an answer is bad, or can it be fooled? We built RubricRobustness, a simple tool that stress tests these graders. We take answers the grader has already scored and deliberately damage them in three ways: flipping statements to mean the opposite, removing parts of the answer, and padding the answer with irrelevant, off-topic text. A trustworthy grader should lower its score whenever an answer is damaged. We tested two widely used grading systems, one for medical questions and one for general questions. The graders reliably caught answers whose meaning had been reversed, but they often failed to penalize answers stuffed with irrelevant filler. Because these graders increasingly decide which AI systems look best, such blind spots matter. We release our tool so anyone can check a grader for these weaknesses before trusting its scores.