Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Text

Reinforcement Learning with Verifiable Rewards (RLVR) has become a cornerstone for unlocking complex reasoning in Large Language Models (LLMs). Yet, scaling up RL is bottlenecked by limited existing verifiable data, where improvements increasingly saturate over prolonged training. To overcome this, we propose Golden Goose, a simple trick to synthesize unlimited RLVR tasks from unverifiable internet text by constructing a multiple-choice question-answering version of the fill-in-the-middle task. Given a source text, we prompt an LLM to identify and mask key reasoning steps, then generate a set of diverse, plausible distractors. This enables us to leverage reasoning-rich unverifiable corpora typically excluded from prior RLVR data construction (e.g., science textbooks) to synthesize GooseReason-0.7M, a large-scale RLVR dataset with over 0.7 million tasks spanning mathematics, programming, and general scientific domains. Empirically, GooseReason effectively revives models saturated on existing RLVR data, yielding robust, sustained gains under continuous RL and achieving new state-of-the-art results for 1.5B and 4B-Instruct models across 15 diverse benchmarks. Finally, we deploy Golden Goose in a real-world setting, synthesizing RLVR tasks from raw FineWeb scrapes for the cybersecurity domain, where no prior RLVR data exists. Training Qwen3-4B-Instruct on the resulting data GooseReason-Cyber sets a new state-of-the-art in cybersecurity, surpassing a 7B domain-specialized model with extensive domain-specific pre-training and post-training. This highlights the potential of automatically scaling up RLVR data by exploiting abundant, reasoning-rich, unverifiable internet text.

Large language models can improve their reasoning skills by practicing on questions where answers can be checked automatically. However, there are not enough high-quality practice problems of this kind, and models often stop improving once they have seen the available data many times. In this paper, we introduce Golden Goose, a simple way to turn ordinary internet text into automatically checkable reasoning tasks. The key idea is to hide an important part of an explanation or solution and ask the model to choose the correct missing piece from several plausible options. This lets us create many new training tasks from useful but previously hard-to-use sources, such as science textbooks, math discussions, coding solutions, and cybersecurity web pages. We show that these new tasks help models continue improving even after they have stopped benefiting from existing training data. The resulting models perform better across a wide range of reasoning benchmarks in math, coding, science, logic, and cybersecurity. Our results suggest that abundant public text can be reused as a scalable source of practice for improving the reasoning ability of AI systems.