Unifying Adversarial Robustness and Training Across Text Scoring Models

Research on adversarial robustness in language models is currently fragmented across applications and attacks, obscuring shared vulnerabilities. In this work, we propose unifying the study of adversarial robustness in text scoring models spanning dense retrievers, rerankers, and reward models. This motivates adapting both attacks and adversarial training methods across model roles. Unlike open-ended generation, text scoring failures are directly testable: an attack succeeds when an irrelevant or rejected text outscores a relevant or chosen one. Using this principled lens of text scoring, we demonstrate that current adversarial training formulations for language models are often short-sighted, failing to effectively generalize across attacks. To address this, we introduce multiple adversarial training methods for text scoring models and show that combining complementary training methods can yield strong robustness while also improving task effectiveness. We also highlight the practical value of our approach for RLHF, showing that our adversarially trained reward models mitigate reward hacking and support the training of better-aligned LLMs. We provide our code and models for further study.

Language models can be fooled by small changes to their inputs, but research on this problem is usually split across different applications and attack types, which hides common weaknesses. We bring these together by focusing on models that assign scores to text, including search models that rank results and reward models that judge LLM responses. In this setting, an attack is simple to define: irrelevant or bad text should not score higher than relevant or good text. We find that common defenses for training more robust models often fail to carry over across attacks, while combining different defensive training methods gives much broader protection without hurting downstream task accuracy, and sometimes even improving it. When used to train LLMs with reinforcement learning, our more robust reward models are also harder to game and lead to better-aligned LLMs.