EAPO: Enhancing Policy Optimization with On-Demand Expert Assistance

Large language models (LLMs) have recently advanced in reasoning when optimized with reinforcement learning (RL) under verifiable rewards. Existing methods primarily rely on outcome-based supervision to strengthen internal LLM reasoning, often leading to inefficient exploration and sparse rewards. To mitigate this issue, we propose Expert-Assisted Policy Optimization (EAPO), a novel RL framework that enhances exploration by incorporating multi-turn interactions with external experts during training. Unlike prior methods, where policies reason in isolation, EAPO incentivizes the policy to adaptively determine when and how to consult experts, yielding richer reward signals and more reliable reasoning trajectories. External assistance ultimately internalizes expert knowledge into the policy model, amplifying the model’s inherent reasoning capabilities. During evaluation, the policy model has been well-optimized to solve questions independently, producing improved reasoning paths and more accurate solutions. On AIME 2024/2025 and AIMO 2025, EAPO consistently outperforms expert-assisted, expert-distilled, and RL baselines, averaging a 5-point gain over self-exploration RL, and also generalizes to non-math benchmarks, including HumanEval, HLE, GPQA, MMLU, EvalPlus, HotpotQA, and SimpleQA.

Large language models can solve many difficult problems, but they often struggle when a task requires careful reasoning over multiple steps. This paper introduces a training method that helps a model learn from stronger expert models only when help is useful. During training, the model can ask experts for guidance, learn from their suggestions, and gradually become less dependent on them. After training, the model answers questions independently without needing any expert assistance. Experiments show that this approach improves performance on challenging math problems and also brings benefits to coding, scientific reasoning, and question-answering tasks. Overall, the method helps language models develop stronger and more reliable reasoning skills through temporary expert guidance.