Benign Overfitting in Adversarial Training for Vision Transformers

Despite the remarkable success of Vision Transformers (ViTs) across a wide range of vision tasks, recent studies have revealed that they remain vulnerable to adversarial examples, much like Convolutional Neural Networks (CNNs). A common empirical defense strategy is adversarial training, yet the theoretical underpinnings of its robustness in ViTs remain largely unexplored. In this work, we present the first theoretical analysis of adversarial training under simplified ViT architectures. We show that, when trained under a signal-to-noise ratio that satisfies a certain condition and within a moderate perturbation budget, adversarial training enables ViTs to achieve nearly zero robust training loss and robust generalization error under certain regimes. Remarkably, this leads to strong generalization even in the presence of overfitting, a phenomenon known as benign overfitting, previously only observed in CNNs (with adversarial training). Experiments on both synthetic and real-world datasets further validate our theoretical findings.

Vision Transformers (ViTs) are the "eyes" of modern AI, but they are easily confused by "adversarial attacks"—tiny, invisible image tweaks that cause failures. While training models to resist these attacks usually makes them less accurate on normal photos, our research proves this doesn't have to be the case. We provide the first mathematical explanation of how ViTs can overfit their training data while remaining robust and accurate on new images—a phenomenon called "benign overfitting." By analyzing the model’s "attention" mechanism, we identified the specific signal-to-noise ratios and maximum perturbation limits required for the model to stay effective under attack. This discovery offers practical guidelines for building AI that is both secure and high-performing, providing key theoretical insights for better adversarial training.