NITP: Next Implicit Token Prediction for LLM Pre-training

Standard Next-Token Prediction (NTP) supervises language models solely through discrete labels in the output logit space. We argue that this sparse, one-hot supervision leaves the latent representation space under-constrained, allowing hidden states to drift into degenerate and anisotropic configurations that limit generalization. To address this issue, we propose Next Implicit Token Prediction (NITP), which augments discrete prediction with dense, continuous supervision directly in the representation space. NITP requires the model to predict the implicit semantic content of the next token, using shallow-layer representations from the same model as stable self-supervised targets. Theoretically, we show that NITP regularizes the optimization landscape by mitigating under-constrained degrees of freedom and enforcing a compact, structured representation geometry. Empirically, across dense and MoE models ranging from 0.5B to 9B parameters, NITP consistently {improves downstream performance with negligible computational overhead}. Notably, on the 9B MoE model, NITP achieves a 5.7\% absolute improvement on MMLU-Pro, along with gains of 6.4\% on C3 and 4.3\% on CommonsenseQA, with $\sim$2\% additional training FLOPs and no additional inference cost. Our implementation is available at https://github.com/aHapBean/NITP.

Large language models learn by repeatedly predicting the next word in a sentence. While effective, this training approach only checks whether the model gets the right answer — it never directly supervises how the model internally organizes its knowledge. We found that this lack of guidance causes the model's internal representations to collapse into a narrow, inefficient structure, limiting its ability to generalize. To fix this, we developed a method called Next Implicit Token Prediction (NITP), which adds a simple requirement during training: the model must also predict the meaning of the next word in its internal space, not just its identity. This acts like a geometric anchor that keeps the model's internal knowledge well-organized. NITP consistently improves model performance across a range of tasks including reasoning, reading comprehension, and knowledge retrieval, while adding less than 2% extra training cost and zero extra cost when the NITP is used.