Procedural Pretraining: Warming Up Language Models with Abstract Data

Pretraining language models directly on web-scale corpora is the de facto paradigm. We study an alternative where the model is initially exposed to abstract structured data to ease the subsequent acquisition of rich semantic knowledge, much like humans learning simple logic and mathematics before higher reasoning. We focus on procedural data, generated by formal languages and other simple algorithms, as such abstract data. We first diagnose the algorithmic skills that different forms of procedural data can improve, often significantly. For example, the accuracy of context recall (Needle-in-a-haystack) jumps from 10 to 98% when a model is pretrained on Dyck sequences (balanced brackets). Second, we study how these gains are reflected in pretraining larger models (up to 1.3B). We find that front-loading as little as 0.1–0.3% procedural data significantly outperforms standard pretraining on natural language, code, and informal mathematics (C4, CodeParrot, and DeepMind-Math datasets). Notably, this also enables the models to reach the same loss value with only 55/67/86% of the original data and thus a comparable reduction in FLOPs. Third, we explore the mechanisms behind the benefits and find that procedural pretraining instills non-trivial structure in both attention and MLP layers. The former is particularly important for structured domains (e.g. code), and the latter for language. Finally, we lay a path for combining multiple forms of procedural data. Our results show that procedural pretraining is a simple, lightweight means of improving performance and accelerating language model pretraining, ultimately suggesting the promise of disentangling knowledge acquisition from reasoning in LLMs.

Language models are typically trained directly on massive collections of internet text. However, this means they must learn factual knowledge and reasoning skills at the same time, which may be inefficient. Humans, by contrast, often first learn abstract patterns such as logic, grammar, and basic mathematics before applying them to more complex knowledge. This paper studies whether language models can similarly benefit from first pretraining on simple forms of structured synthetic data. We focus on “procedural data”: sequences generated by formal rules and algorithms, such as balanced brackets or other symbolic patterns. These datasets contain no real-world knowledge, but they can teach models useful abstract skills such as memory, structure recognition, and algorithmic reasoning. Our results show that even a short initial training phase on such data can significantly improve and speed up later pretraining on natural language, code, and mathematics.