Escaping the Verifier: Learning to Reason via Demonstrations

Training Large Language Models (LLMs) to reason often relies on Reinforcement Learning (RL) with task-specific verifiers. However, many real-world reasoning-intensive tasks lack verifiers, despite offering abundant expert demonstrations that remain under-utilized for reasoning-focused training. We introduce **RARO** (Relativistic Adversarial Reasoning Optimization), which learns strong reasoning capabilities from expert demonstrations alone via **Inverse Reinforcement Learning**. Our method sets up an adversarial game between a **policy** and a **relativistic critic**: the policy learns to mimic expert answers, while the critic aims to identify the expert among (expert, policy) answer pairs. Both the policy and the critic are trained jointly and continuously via RL, and we identify key stabilization techniques required for robust learning. Empirically, RARO significantly outperforms strong verifier-free baselines across all evaluation tasks: $+13.7\%$ accuracy on Countdown ($1.5$B), +8.2\% on DeepMath ($7$B), and $+19.1\%$ win-rate on Poetry Writing ($7$B) against expert poems. RARO also exhibits similar robust scaling trends as RL with verifiers. These results demonstrate that RARO effectively elicits strong reasoning performance from expert demonstrations alone, enabling robust reasoning learning even when task-specific verifiers are unavailable.

Many recent AI systems get better at solving hard problems by practicing with automatic checkers that can tell whether each answer is correct, such as math or programming tests. But many important tasks, such as analytical or research assistance, financial analysis, or creative writing, do not have simple answer checkers, even though they often have many examples written by experts. We introduce RARO, a training method that helps language models learn stronger reasoning from these expert examples alone. RARO trains the model to play two roles: one role tries to produce answers like an expert, while the other compares the model's answer with an expert answer and judges which is better. This comparison game gives the model useful feedback without requiring a hand-built checker or new human preference labels. In experiments on arithmetic puzzles, advanced math problems, and poetry writing, RARO consistently improved over existing training methods on expert examples and came close to checker-based training when checkers were available. This suggests that stronger reasoning models can be trained for real-world tasks where quality matters but correctness is hard to check automatically.