Adaptive and interpretable two-sample test as a batch-level tool for AI safety

Most safety defenses for large language models - input and output classifiers, refusal probes, per-query OOD detectors - inspect prompts and completions one at a time. Recent work shows that black-box iterative jailbreaks evade even the strongest of these (Davies et al 2026), arguing that "effective defence requires supplementing single-interaction methods with batch-level monitoring." This suggests treating incoming prompts as a collection and asking whether their distribution deviates from benign traffic - what statisticians call a two-sample test. This is a non-trivial problem: attack queries are heavily diluted by benign traffic, new attacks can circumvent fixed rules, and detection claim must be calibrated. We propose batch-level monitoring based on a calibrated, adaptive two-sample test applied to a model's hidden states, requiring only an attack-free reference. The test returns both an anomaly score and discovered features localizing the attack signal in activation space, making the detection output itself an interpretability output. On llama3-jailbreaks, it detects attacks significantly better than standard two-sample baselines, and reveals that jailbreaks organize into two near-orthogonal groups - scaffolded versus direct prompts - visible in benign data too, contradicting a single refusal-direction picture.