Is your LLM a Sequence Model on the Training History? The Origins and Consequences of Anticipation

Recent work showed that, in cyclic fine-tuning settings, transformers exhibit striking \emph{anticipatory recovery}: the loss on an upcoming document decreases \emph{before} that document is revisited. We extend this phenomenon beyond individual-document cycling to bursty pre-training over mixtures of class-conditional distributions, where examples from the same class appear in predictable bursts -- so the model is trained on one class for several steps, then on the next one, and so on. We show that Transformers anticipate upcoming classes despite having no explicit memory of training history, and that \emph{anticipation is linked to a lower training loss}. In our experiments, models realize a substantial fraction of the anticipatory advantage achievable by sequence models that explicitly learn class order. Additionally, we demonstrate anticipation in a setting where the next class can only be predicted by knowing the previous two classes, suggesting that Transformers can partially behave like sequence models over the training stream itself. To study the optimization dynamics underlying anticipation, we analyze the implicit bias of bursty mini-batch training and identify an implicit alignment pressure between temporally adjacent class gradients. Finally, we show that structured training order reshapes measured relationships between classes, suggesting that training order itself can act as a source of representational bias during optimization.