Krause Synchronization Transformers

Self-attention in Transformers relies on globally normalized softmax weights, causing all tokens to compete for influence at every layer. When composed across depth, this interaction pattern induces strong synchronization dynamics that favor convergence toward a dominant mode, a behavior associated with representation collapse and attention sink phenomena. We introduce $\textbf{Krause Attention}$, a principled attention mechanism inspired by bounded-confidence consensus dynamics. Krause Attention replaces similarity-based global aggregation with distance-based, localized, and selectively sparse interactions, promoting structured local synchronization instead of global mixing. We relate this behavior to recent theory modeling Transformer dynamics as interacting particle systems, and show how bounded-confidence interactions naturally moderate attention concentration and alleviate attention sinks. Restricting interactions to local neighborhoods also reduces runtime complexity from quadratic to linear in sequence length. Empirically, we validate Krause Attention across diverse settings, including vision (ViT on CIFAR/ImageNet), autoregressive image generation (MNIST/CIFAR-10), large language models (Llama/Qwen), and language models trained from scratch at multiple scales (100M/200M). Across these domains, Krause Attention achieves consistent performance gains while improving computational efficiency, highlighting bounded-confidence dynamics as a scalable and effective inductive bias for attention. Project page is available at https://jingkun-liu.github.io/krause-sync-transformers/.

In standard Transformer models (the architecture behind ChatGPT), every piece of data usually pays attention to every other piece. However, as these models get deeper, this "everyone talks to everyone" approach can cause a problem: the distinct features of the data blend together too much, losing important details and focusing on a few patterns. To solve this, we introduce Krause Attention. Inspired by Krause consensus model where people can only talk to those who share similar views, Krause Attention restricts global interactions. Instead of looking at the whole sequence, data points only interact with their opinion-close neighbors. This localized approach helps the model maintain rich and structured information, reducing model’s processing time drastically to linear in sequence length. We tested Krause Attention across various image and language models and found that it consistently makes AI models both faster and smarter.