Attention Sinks in Diffusion Transformers: A Causal Analysis

Attention sinks---tokens that receive disproportionate attention mass---are assumed to be functionally important in autoregressive language models, but their role in diffusion transformers remains unclear. We present a causal analysis in text-to-image diffusion, dynamically identifying dominant attention recipients per timestep and suppressing them via paired, training-free interventions on the score and value paths. Across 553 GenEval prompts on Stable Diffusion~3 (with SDXL corroboration), removing these sinks does not degrade text-image alignment (CLIP-T) or preference proxies (ImageReward, HPS-v2) at $k{=}1$; only under stronger interventions ($k\!\geq\!10$) does HPS-v2 exhibit a metric-dependent boundary, while CLIP-T remains robust throughout. The perceptual shifts induced by suppression are nonetheless \emph{sink-specific}---$\sim\!6\times$ larger than equal-budget random masking---revealing an empirical dissociation between trajectory-level perturbation and \emph{semantic alignment} in diffusion transformers. \footnote{Code available at \url{https://github.com/wfz666/ICML26-attention-sink}.}

Modern image generation systems use attention mechanisms to decide how different parts of a prompt and image influence one another. A common intuition is that the tokens receiving the most attention are especially important, and therefore should be protected when making these models faster or easier to analyze. We tested this intuition in diffusion transformers, a family of models behind many recent text-to-image systems. Instead of only observing where attention goes, we directly intervened during image generation: at each denoising step, we identified the tokens receiving unusually high attention and suppressed their contribution. Surprisingly, removing these high-attention “sink” tokens did not reduce standard measures of text-image alignment or preference under normal settings. However, the generated images often changed visually, and these changes were much larger than those caused by removing random tokens. Our findings show that high attention does not necessarily mean semantic necessity. In diffusion transformers, dominant attention recipients can shape the visual path of generation without being required for preserving the prompt’s meaning. This cautions against treating attention mass alone as a reliable signal of importance in efficient or interpretable image generation systems.