Plan Then Action: High-Level Planning Guidance Reinforcement Learning for LLM Reasoning

Large language models (LLMs) demonstrate strong reasoning abilities via Chain-of-Thought (CoT), but their token-level generation encourages local decisions and lacks global planning, often leading to redundant or inaccurate reasoning. Existing methods, such as tree-based search and reinforcement learning (RL), attempt to address this issue but incur high computational costs and still struggle to produce reliable reasoning trajectories. To address these challenges, we propose Plan-Then-Action Enhanced Reasoning with Group Relative Policy Optimization (PTA-GRPO), a two-stage framework designed to jointly improve high-level planning and fine-grained CoT reasoning. Specifically, in the first stage, a given LLM is responsible for summarizing CoT reasoning into compact high-level guidance, which is then leveraged for supervised fine-tuning. Then, we introduce a guidance-aware reinforcement learning method that jointly optimizes the final output and the quality of guidance, enhancing reasoning effectiveness. We evaluate PTA-GRPO on ten reasoning benchmarks across mathematics and natural sciences, using five diverse base models spanning multiple data modalities. The results show that PTA-GRPO consistently delivers stable and significant improvements across models and tasks, demonstrating strong effectiveness and generalization.

Large language models are increasingly able to solve complex problems in mathematics, science, and other reasoning tasks. However, they often generate answers step by step without first forming a clear overall plan. This can lead to unnecessary reasoning, inconsistent steps, or mistakes that are carried through the rest of the solution. In this work, we propose a training method that teaches language models to first make a concise high-level plan and then use that plan to guide their detailed reasoning. The method also encourages the model to produce plans that are useful for reaching correct answers, rather than only rewarding the final answer itself. Experiments on a wide range of reasoning tasks show that this approach improves performance across different models and benchmarks. These results suggest that helping AI systems plan before acting can make their reasoning more reliable, organized, and effective.