Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for Open-Ended LLM Reasoning

Recent advances in Large Language Models (LLMs) have underscored the potential of Reinforcement Learning (RL) to facilitate the emergence of reasoning capabilities. Despite the encouraging results, a fundamental dilemma persists as RL improvement relies on learning from high-quality samples, yet the exploration for such samples remains bounded by the inherent limitations of LLMs. This, in effect, creates an undesirable cycle in which what cannot be explored cannot be learned. In this work, we propose Rubric-Scaffolded Reinforcement Learning (RuscaRL), a novel instructional scaffolding framework designed to break the exploration bottleneck for open-ended reasoning. Specifically, RuscaRL introduces checklist-style rubrics as (1) explicit scaffolding for exploration during rollout generation, where different rubrics are provided as external guidance within task instructions to steer diverse high-quality responses. This guidance is gradually decayed over time, encouraging the model to internalize the underlying reasoning patterns; (2) verifiable rewards for exploitation during model training, where we can obtain robust LLM-as-a-Judge scores using rubrics as references, enabling effective RL on open-ended reasoning tasks. Extensive experiments demonstrate the superiority of the proposed RuscaRL across various benchmarks, effectively expanding reasoning boundaries under the Best-of-N evaluation.

Modern AI language models can solve many problems, but they often improve only when they already know how to produce good examples to learn from. This creates a bottleneck: for open-ended tasks such as medical advice or creative writing, there may be no single correct answer, and the model may not discover better ways to respond on its own. We propose RuscaRL, a training method that gives the model temporary guidance in the form of simple checklists describing what a good answer should include or avoid. During training, these checklists help the model try more useful and diverse answers, and the guidance is gradually removed so the model learns to respond well without relying on it. The same checklists are also used to score the model’s answers more consistently. Experiments across several open-ended benchmarks show that this approach helps models produce higher-quality responses than standard training methods. The results suggest that carefully designed checklists can make AI training more effective for tasks where quality is hard to measure with a single right-or-wrong answer.