Sample-Efficient Diffusion-based Reinforcement Learning with Critic Guidance

Recent advances in reinforcement learning (RL) have achieved great successes by leveraging the multimodality and exploration capability of diffusion policies. Among these approaches, one representative branch focuses on the sampling-based policy optimization. This design enables better exploration capability of the diffusion model, particularly at the beginning of training, but suffer from low exploitation in Q-value information, resulting in a slow policy convergence. Another branch pays attention to gradient-based policy optimization, which sufficiently exploits the gradient of the Q function yet tends to collapse into a unimodal policy with low diversity. To address this issue, we propose CGPO, \textbf{C}ritic-\textbf{G}uided diffusion \textbf{P}olicy \textbf{O}ptimization, which effectively balances exploration and exploitation with the training-free guidance technique integrated into the denoising process of diffusion policy. Concretely, CGPO steers action generation toward high-value regions defined by the critic network and uses the guided actions as regression objectives. In this manner, CGPO reduces the time required to obtain high-quality actions and improves final performance with better balance between the exploration-exploitation tradeoff. We validate the effectiveness of CGPO on 5 MuJoCo locomotion tasks, and CGPO achieves state-of-the-art performance compared with existing diffusion-based RL methods. Notably, CGPO is the first success to incorporate diffusion policy into real-world RL, with its superior performance on Franka robot arm grasping tasks. Our official page is released at \url{https://dingsht.tech/cgpo-webpage}.

Diffusion policies have recently shown strong performance in robotic control, particularly in imitation learning, where they model complex and multimodal actions from human demonstrations. However, collecting such data is costly and limits performance to human expertise. Diffusion-based reinforcement learning aims to remove this dependency, but existing methods often rely on inefficient action sampling or unstable value guidance, leading to slow improvement and limited effectiveness in real-world robotic tasks. We propose CGPO, a diffusion-based reinforcement learning framework that directly integrates critic guidance into the action generation process. Unlike prior methods that depend on sampling large numbers of candidate actions or loosely guiding policies with value gradients, CGPO steers action generation toward high-value regions during denoising. This results in more efficient and precise policy updates, while maintaining sufficient exploration. Importantly, CGPO is practical for real-world deployment. It avoids expensive sampling procedures and does not require human demonstrations, enabling effective learning directly from interaction. We demonstrate that CGPO consistently improves performance in both simulation and real-world robotic tasks, highlighting its potential for scalable robot learning.