Modelling Attention with Aitchison Geometry: Token Distinguishability and Temperature Scaling

The attention mechanism with softmax normalisation is a foundational component of Transformer-based large language models. However, with very long contexts, attention scores are known to diminish, raising fundamental questions about token distinguishability and how it can be preserved. In this work, we provide a formal characterisation of token distinguishability in attention as a function of context length and embedding dimension. We introduce Aitchison distance to quantify relative differences among attention probabilities, and show that, with Gaussian queries and keys, even in the long-context regime, token distinguishability converges to a finite, non-zero limit rather than vanishing. Leveraging the linear relationship between inverse-temperature scaling and Aitchison distance, we derive a theoretical lower bound of $\Omega(\sqrt{\log L})$ on the logit scaling required to produce a sharp attention distribution. Finally, we demonstrate that Aitchison distance provides a principled and practical alternative to entropy for monitoring training and inference, as it captures the full compositional structure, including the smaller components of the attention probabilities.

Long data passages are a challenge for large language models or AI models because they must decide which parts of the input are most important for the output. These "important scores" become much smaller as the data gets longer, making it difficult to tell whether the model is making useful choices or losing focus on the important pieces of information. We study this problem by measuring the importance scores relative to each other. This helps tell the difference between the scores shrinking because of the data's size or because the model is no longer making clear choices. We show that, in idealistic conditions, these relative differences do not disappear just because the input is long. However, in non-idealistic, real-world conditions, we show the strength of the adjustment to the scores needed to help the model make clear choices. We also show that measuring the relative scores provides important information in the real-world development and deployment of AI models.