Universal One-third Time Scaling in Learning Peaked Distributions

Training large language models (LLMs) is computationally expensive, partly because the loss exhibits slow power-law convergence whose origin remains debatable. Through systematic analysis of toy models and empirical evaluation of LLMs, we show that this behavior can arise intrinsically from the use of softmax and cross-entropy. When learning peaked probability distributions, e.g., next-token distributions, these components generically yield power-law vanishing losses and gradients, regardless of many microscopic details, creating a fundamental optimization bottleneck. This ultimately leads to power-law time scaling of the loss with a universal exponent of $1/3$. Our results provide a mechanistic explanation for observed neural scaling and suggest new directions for improving LLM training efficiency.

Training large language models (LLMs) like ChatGPT requires enormous amounts of computation, partly because performance improves only very slowly with more training, following a "power law" pattern whose origins were poorly understood. Knowing why this slowdown occurs is essential for designing more efficient AI systems. We discovered that this sluggishness is not an accident of the data or the scale of the model, but is baked into two core architectural ingredients used in virtually every modern language model: the softmax function and the cross-entropy loss. When a model learns to predict the next word in a sentence, it must output a sharply peaked probability distribution over its entire vocabulary. We show, through mathematical analysis and experiments on toy models, that softmax and cross-entropy together make gradients vanish as a power law in this regime, creating a fundamental bottleneck. This leads to a universal training slowdown where loss decreases as time to the power of negative one-third, regardless of many other details of the model or data. We validate this prediction on open-source language models across a range of sizes. Our findings explain a long-standing empirical mystery and point toward concrete strategies for faster LLM training, such as redesigned output layers that avoid the softmax bottleneck altogether.