A geometric relation of the error introduced by sampling a language model's output distribution to its internal state

GPT-style language models are sensitive to single-token changes at generation points where the predicted probability distribution is spread across multiple tokens. Viewing this sensitivity as a geometric property, we derive an $\mathfrak{so}(n)$-valued 1-form that depends only on the geometry of the token embeddings. Despite this purely geometric origin, we show that its curvature is semantically meaningful: on chess reasoning tasks, the curvature couples to the world model of an off-the-shelf instruction-tuned model, with transformations clustering by board region and respecting piece importance. Our findings suggest that token space geometry directly reflects how models internally represent problems.

AI chatbots work by generating one word at a time. They do this by assigning probabilities to each possible word at every time step. At most time steps, the chatbots are confident, and so only one word out of the many possible ones is very probable. However, there are some time steps where this is not the case, and the chatbot is conflicted between a few possible next words. These conflicted moments can be very significant, changing the overall answer to a reasoning problem, despite corresponding to only a small change between two similar words. The confidence of the chatbot is usually summarised in a single number, but we are able to show that there is more structure here. By making a particular geometric construction, we are able to relate these uncertain moments to how the chatbot internally conceptualises high-level strategic elements of its task. We use ordinary commercial AI chatbots and make them think through chess positions to analyse this.