WARP: Measuring and Mitigating Evaluation Awareness in Browser-Agent Safety Benchmarks

Browser-agent safety benchmarks underwrite deployment safety claims, but only if model behavior under evaluation generalizes to deployment. We present the first cross-benchmark measurement of evaluation awareness in browser-agent indirect prompt-injection (IPI) benchmarks, across six frontier models and four published benchmarks (DoomArena, WASP, AgentDojo, InjecAgent; n=4,608 trajectories). When asked to classify trajectory purpose, models read over half of trajectories on every benchmark as evaluations, and three of four benchmarks elicit unprompted awareness verbalization. Awareness has a causal effect on reported attack-success rate (ASR): adding an explicit evaluation cue drops ASR by 10.9 pp on DoomArena, while a length-matched non-eval cue does not. A residual-stream probe on an open-weight model also separates attacked from safe trajectories at ≈2σ. Together, these results indicate that frontier models systematically distinguish IPI safety benchmarks from deployment, affecting reported ASR validity. Building upon this diagnosis, we release WARP (Wrapper-based, Adaptive, Realistic Pipeline), a benchmark-generation pipeline that builds challenging and realistic IPI evaluation tasks on top of any seedable synthetic browser environment. Instantiated as a 50-task benchmark on WebArena Verified, WARP attains 80.7% headline ASR across the same six models, substantially above raw DoomArena (22.3%) and WASP (27.0%) rates, while lowering eval-awareness rates relative to a no-rewrite baseline. We advocate routine reporting of awareness. Code, benchmark, and the awareness-measurement toolkit are released.