Balancing Fidelity and Diversity in Diffusion Models via Symmetric Attention Decomposition: Hopfield Perspective

We characterize the pre-softmax attention matrix $\mathbf{QK^\top}$ in transformers as an associative memory matrix encoding pairwise associations between input features. By decomposing this matrix into its symmetric and skew-symmetric parts, we interpret the symmetric component as governing the structure of the _energy landscape_, and the skew-symmetric component as driving _circulation_ on that landscape. Leveraging the energy formulation induced by the symmetric component, we derive Hopfield-style stability measures that quantify the stability of retrieved features. We observe meaningful correlations between Hopfield-style stability measures and the fidelity--diversity trade-offs in generation. Finally, we propose a controllable knob to modulate this trade-off by modifying the circulation of the underlying dynamics. Code is available at our Project Page (https://hyeon-cho.github.io/Balancing/).

AI image generators can create rich and detailed pictures, but they sometimes produce strange errors, such as an animal with three legs or objects whose shapes and textures are incorrectly blended. These errors often come from attention, a key mechanism that lets different parts of an image influence one another during generation. Attention is useful because it helps the model connect related visual pieces into a coherent scene. However, the same process can also create wrong connections, making it difficult to separate helpful interactions from harmful ones. We study attention as a kind of associative memory, where one visual piece can bring related pieces to mind. From this view, attention is not only a way to combine image parts, but also a map of how those parts interact. We show that this map can be split into two components: one encourages stable, coherent structures, while the other creates directional movement that can help the model escape confused mixtures. By adjusting this movement, we can sometimes repair images that are stuck in flawed configurations, while too much adjustment can disturb images that were already well formed. This perspective helps explain why attention can both improve image coherence and produce visual confusion. It may also help make future image generators easier to understand, adjust, and trust.