Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA

Large Language Models (LLMs) obey consistent scaling laws---empirical power-law fits that predict how loss decreases with compute, data, and parameters. While predictive, these laws are descriptive rather than prescriptive: they characterize typical training, not optimal training. Surprisingly few works have successfully challenged the data-efficiency bounds implied by these laws---which is our primary focus. To that end, we introduce the Geodesic Hypothesis, positing that token sequences trace geodesics on a smooth semantic manifold and are therefore locally linear. Building on this principle, we propose a novel Semantic Tube Prediction (STP) task, a JEPA-style regularizer that confines hidden-state trajectories to a tubular neighborhood of the geodesic. STP generalizes JEPA to language without requiring explicit multi-view augmentations. We show this constraint improves signal-to-noise ratio, and consequently preserves diversity by preventing trajectory collisions during inference. Empirically, STP allows LLMs to match baseline accuracy with 16$\times$ less training data, directly violating the data term of fine-tuning scaling laws and demonstrating that principled geometric priors can surpass brute-force scaling. Code: \url{https://github.com/galilai-group/llm-jepa#stp}

Large language models usually improve by training on more data, using more computing power, or increasing the number of model parameters. This paper asks whether there is a better way: can we make models learn more efficiently from the same amount of data? We introduce Semantic Tube Prediction (STP), a new training method based on the idea that the internal steps a language model takes while processing text should follow a smooth and consistent path. Instead of only training the model to predict the next word, we also encourage its internal representations to move in a more orderly way. This reduces unnecessary noise during learning and helps the model preserve meaningful differences between examples. Across several datasets, model families, and model sizes, this extra training signal improves accuracy and data efficiency. In our experiments, models trained with STP can match the performance of standard training while using up to 16 times less training data. The method is simple to add to existing language model training, adds little overhead, and is not needed during inference. These results suggest that better training principles, not just larger datasets or larger models, can substantially improve how efficiently language models learn.