Same Question, Different Lies: Cross-Context Consistency (C³) for Black-Box Sandbagging Detection

As language models grow more capable, accurate capability evaluation becomes essential for safety decisions. If models can deliberately underperform on dangerous capability evaluations—a behavior known as **sandbagging**—they may evade safety measures designed for their true capability level. We introduce Cross-Context Consistency ($C^3$), a general framework for unsupervised black-box sandbagging detection that exploits a fundamental asymmetry: when a model truly lacks capability, its confusion manifests consistently across paraphrased questions, but when a capable model feigns incompetence, its strategic choices about *how* to appear weak create detectable inconsistencies. The framework is agnostic to the specific consistency metric and aggregation method; we present a simple instantiation using embedding cosine distance and mean aggregation that requires no training data or model-specific adaptation. We evaluate $C^3$ across prompted and fine-tuned sandbagging scenarios, across instructed and more naturalistic setups, maintaining a classification signal where other black-box methods fail. Our findings show the limitations of existing sandbagging detection methods, and reveal the efficacy of consistency-checking as a detection mechanism for sandbagging.

Before powerful AI systems are released, developers run safety tests to check what they can do. But what if an AI deliberately fails these tests to avoid safety restrictions, then reveals its true abilities once deployed? This behaviour, called "sandbagging," could let dangerous capabilities slip past the very checks designed to catch them. Existing detection methods often require access to a model's internal workings, which we cannot count on having. We developed Cross-Context Consistency (C³), a simple method that detects sandbagging by asking the same question multiple ways. The intuition is like an employee who deliberately underperforms to avoid being handed more work: they want to look just capable enough to keep their job, but not so capable that they get loaded up with extra responsibilities. A colleague who is genuinely struggling trips up on the same things every time, because real limitations are stable and predictable. But someone who's only pretending has to actively decide, each time, where to hold back. And when the same task reappears in a slightly different form, those choices don't line up. C³ measures these inconsistencies to flag when an AI is performing below its true ability. It requires no access to the model's internals, no examples of sandbagging to learn from, and no system-specific customisation. We tested C³ across a range of sandbagging scenarios and showed that it reliably detected fake incompetence even where existing methods failed, offering a practical tool to help ensure AI safety evaluations can be trusted.