Attention Implements the Fisher Geometry of Exponential Families

Softmax attention normalizes scores, and Bayes’ rule normalizes log prior plus log likelihood. For finite latent symbols with exponential-family observations, we show that one attention head can implement the Bayes posterior and posterior means exactly, and that the posteriors representable by a single head are precisely log-linear. The standard exponential-family duality identity rewrites the likelihood as a negative Bregman divergence in mean/sufficient-statistic space; our attention-specific contribution is to use this identity to characterize when Bayes-aligned attention admits one globally shared quadratic metric, proving that this happens exactly when the dual potential is quadratic. When curvature varies, we give a multi-head local-curvature atlas with approximation and head-count bounds, and we extend the picture to in-context estimation through plug-in consistency, finite-sample stability, and an optimizer-agnostic converse from excess log-loss to approximate key-subspace alignment. Controlled Gaussian, Bernoulli, and Poisson ICE diagnostics illustrate these regimes, while the exact theorems remain scoped to finite discrete latent classes and suggest testable, not universal, predictions for larger learned transformers.

Modern transformer AI systems rely on attention: a mechanism that scores which examples or pieces of information are most relevant. This paper asks when those scores can act like correct statistical reasoning from noisy data. We study a clean setting where the hidden answer is one of several categories and each observation is noisy. We show that a single attention head can exactly reproduce the ideal rule for updating beliefs from evidence in a broad class of such problems. We also show an important limit: one fixed notion of similarity is only truly correct in simple, Gaussian-like cases. In more complicated noise models, the right notion of similarity changes across the problem, so multiple attention heads can be understood as using several local “yardsticks.” This gives mathematical guidance for when one attention head is enough and when several heads are needed. In synthetic experiments, trained attention models behave as the theory predicts: they approach the ideal statistical rule as they receive more examples, and their learned similarity matches the expected noise structure in the Gaussian case.