Representational Curvature Modulates Behavioral Uncertainty in Large Language Models

Temporal straightening describes how, across layers in autoregressive LLMs, the trajectory traced by the token representations of an input sequence becomes straighter, potentially enabling next-token prediction by linear extrapolation. However, a direct link between this trajectory and token-level behavior has been missing. We provide such a link by relating contextual curvature—a geometric measure of how sharply the representational trajectory bends over recent context—to next-token entropy. Across two models (GPT-2 XL and Pythia-2.8B), contextual curvature is correlated with entropy, and this relationship emerges during training. Perturbation experiments reveal selective dependence: manipulating curvature through trajectory-aligned interventions reliably modulates entropy, while geometrically misaligned perturbations have no effect. Finally, regularizing representations to be straighter during training modestly reduces token-level entropy without degrading validation loss. These results identify trajectory curvature as a task-aligned representational feature that influences behavioral uncertainty in LLMs.

When a language model processes a sentence, it converts each word into a point in a space with thousands of dimensions. The sentence then traces a path through that space, one point per word. Earlier work showed that during training, models with multiple layers of processing learn to straighten these paths in their middle layers. Hypothetically, the model does this because it is easier to predict the next item in a trajectory when the trajectory is straighter. In this work, we demonstrate that there is indeed a relationship between this straightening and how the model chooses the next token. We measured how sharply each trajectory bends just before a given word—its curvature—and asked whether bendier paths go together with less confident next-word guesses. In two different language models, they do, and the link emerges gradually over training. Nudging the model's state along the direction the path is already moving reliably changes its prediction confidence; nudging in unrelated directions does not. Adding a small training penalty that encourages straighter paths slightly reduces prediction uncertainty without hurting accuracy. The result ties an internal geometric feature of language models to their confidence, supporting the idea that prediction works by straightening and offering a new tool for shaping how certain a model is.