Context-level Language Modeling by Learning Predictive Context Embeddings

We propose ContextLM, a framework that implicitly learns multi-token prediction by augmenting standard pretraining with an intrinsic next-context prediction objective. ContextLM builds a language model on top of context embeddings that span multiple tokens, enabling better next-token prediction by predicting the next context. Our model is fully compatible with standard autoregressive, token-by-token evaluation paradigms (e.g., perplexity). Extensive experiments with GPT-2 and Pythia backbones (up to 1.5B parameters and 300B training tokens) reveal that ContextLM shifts the Pareto frontier of scaling laws, exhibiting superior efficiency in parameters, training tokens, and FLOPs. Our results show that ContextLM could already achieve the baseline perplexity using 39\% fewer parameters and demonstrates robust generalization improvements on extensive downstream tasks under equivalent parameter counts.

We introduce ContextLM, a hierarchical language modeling framework that learns predictive, context-level representations while remaining fully compatible with standard token-level evaluation metrics. By introducing a Context Predictor that autoregressively models higher-level context embeddings, our approach enables multi-token prediction implicitly and leverages aggregated error signals from future tokens to capture long-range dependencies. Experiments on the GPT-2 and Pythia families, up to $1.5$B parameters, show consistent improvements in perplexity and downstream performance, with gains persisting across model scales. Further analysis demonstrates that context-level supervision strengthens long-range coherence and attention allocation, highlighting next-context prediction as a scalable and efficient direction for advancing large language models.