Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction

Autoregressive models (ARMs) currently constitute the dominant paradigm for large language models (LLMs). Energy-based models (EBMs) represent another class of models, which have historically been less prevalent in LLM development, yet naturally characterize the optimal policy in post-training alignment. In this paper, we present a unified view of these two model classes. Taking the chain rule of probability as a starting point, we establish an explicit bijection between ARMs and EBMs in function space, which we show to correspond to a special case of the soft Bellman equation in maximum entropy reinforcement learning. Building upon this bijection, we derive the equivalence between supervised learning of ARMs and EBMs. Furthermore, we analyze the distillation of EBMs into ARMs by providing theoretical error bounds. Our results provide insights into the ability of ARMs to plan ahead, despite being based on the next-token prediction paradigm.

Right now, most popular AI chatbots work by simply guessing the next word in a sequence, one word at a time. There is another, less common way to build AI that looks at entire responses at once to determine the absolute most ideal answer. This paper demonstrates a direct mathematical link between these two completely different approaches, proving they are actually just two sides of the same coin. It shows that an AI using the step-by-step method can be perfectly translated into the big-picture method, and vice versa. Because these two methods are mathematically equivalent, teaching an AI using the simpler step-by-step method can theoretically achieve the exact same result as teaching it with the more complex big-picture method. Ultimately, this paper explains why chatbots that only guess the next word can seem so intelligent, proving that by learning to predict the next word perfectly, the AI is inherently acting as if it had planned out its entire response in advance.