PretrainZero: Reinforcement Active Learning on Pretraining Data

Recent reinforcement learning (RL) based large-thinking models demonstrate impressive expert-level abilities, i.e., software and math, but still rely heavily on verifiable rewards in specific domains, which places a significant bottleneck to extend the performance boundary of general reasoning capabilities. In this work, we propose PretrainZero, a reinforcement active learning framework built on the pretraining corpus to extend RL from domain-specific post-training to general pretraining. PretrainZero features the following characteristics: 1) Active pretraining objectives: inspired by the active learning ability of humans, PretrainZero learns a unified reasoning policy to actively identify reasonable and informative contents from pretraining corpus, and reason to predict these contents by RL. 2) Self-supervised learning: without any verifiable labels, pretrained reward models, or supervised fine-tuning, we directly pretrain reasoners from $3\sim30$B base models on the general Wikipedia corpus using RL, significantly breaking the verification data-wall for general reasoning. 3) Verification scaling: by tackling increasingly challenging masked spans, PretrainZero substantially enhances the general reasoning abilities of pretrained base models. With reinforcement learning on pretraining data, PretrainZero improves Qwen3-4B-Base for 8.43, 5.96 and 10.60 on MMLU-Pro, SuperGPQA and math average benchmarks. In post-training, the pretrained models can also serve as reasoning foundation models for downstream RLVR tasks.

Today’s most capable reasoning language models can solve difficult mathematics and programming problems, but they usually improve by practicing on questions whose answers can be automatically checked. This dependence limits their ability to learn broader reasoning skills from the large amount of general text available online. We introduce PretrainZero, a method that teaches language models to practice reasoning while reading ordinary pretraining text, such as Wikipedia. Instead of requiring human-written solutions or specialized checking systems, the model learns to actively identify informative parts of a passage, temporarily hide them, and reason about what information is missing. As training progresses, it learns from increasingly challenging missing-content prediction tasks.