Attacking the Representation Manifold: A Mechanistic Study of Adversarial Robustness in Modular Addition

Neural networks trained on modular addition learn algorithms whose latent representations factor through a torus-to-circle map, providing unusually complete knowledge of the learned algorithm and its representation geometry. We exploit this transparency to show how mechanistic knowledge allows us to predict the form of successful adversarial perturbations and how adversarial training reshapes representations to resist attack. We decompose adversarial perturbations on the embedding torus into phase-shifting and amplitude-changing components, predicting that efficient attacks target the same Fourier features the model uses. We confirm this empirically: the Fourier spectrum of successful PGD perturbations concentrates on the model's frequency features, mechanism-informed attacks restricted to those frequencies are competitive with white-box PGD, and attack transfer between models is predicted by their feature overlap. The same mechanistic lens predicts that adversarial training increases robustness by broadening the model's frequency support, linking the representation change to capacity-robustness trade-offs. Modular addition thus provides a case study in which adversarial vulnerability becomes interpretable - vulnerability becomes a targeted failure of the learned algorithm, and robustness becomes a measurable restructuring that algorithm.