Attention's forward pass and Frank-Wolfe

We analyze the hardmax limit of self-attention dynamics for token embeddings in the zero-temperature regime $(\beta \to +\infty)$ and relate it to finite-$\beta$ behavior. In this limit, the update rule can be viewed as a Frank-Wolfe step for a quadratic objective over the convex hull of the current tokens. When the key-query matrix is negative semidefinite, the dynamics converge with the standard sublinear rate $\mathcal{O}(t^{-1})$ on the quadratic energy, whereas in the positive semidefinite case, extending the hardmax rule to the convex hull induces a Voronoi structure: vertices are stationary, interior points remain in their initial cells, and each token moves along a straight line toward its cell's vertex with exponential convergence under a step-size bounded away from zero. We additionally establish well-posedness of the associated ODE limit in this regime. For finite $\beta$, we model self-attention as a Markov chain and prove *dynamic metastability*: interior tokens reach near-vertex configurations in a constant number of steps and remain trapped for times exponential in $\beta$ with high probability, before eventual collapse to some point within the initial convex hull. Thus, hardmax dynamics accurately approximate the finite-$\beta$ process over exponentially long time horizons.

Transformers, the models behind many modern language systems, decide which words or pieces of text should influence each other through a process called attention. This paper studies a simplified version of attention where this influence becomes very focused: instead of each word being affected by many others, it mostly follows the one it is most strongly connected to. In this setting, the words can be imagined as points inside the smallest “outline” that contains them all, like stretching a rubber band around a group of pins on a board. The paper shows that attention tends to move words toward special points on this outline. It also shows that the usual, smoother form of attention can quickly enter long-lasting patterns before eventually merging together. This is called metastability: intuitively, it is like a ball on a very gently curved hillside, where the slope is so slight that the ball seems almost stuck and moves only very slowly, but over time it still drifts sideways and slides away. In this paper, metastability means that words can remain organized around a few especially influential words for a long time before the system finally settles into a more collapsed state. By explaining when and why these long-lasting patterns appear, the theory gives a clearer picture of how attention organizes information over time, and may help researchers better understand which words become influential and how complex behavior can arise from simple attention rules.