From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuning

We introduce Distribution Contractive Reinforcement Learning (DICE-RL), a framework that uses reinforcement learning (RL) as a "distribution contraction" operator to refine pretrained generative robot policies. DICE-RL turns a pretrained behavior prior into a high-performing "pro" policy by amplifying high-success behaviors from online feedback. We pretrain a diffusion- or flow-based policy for broad behavioral coverage, then finetune it with a stable, sample-efficient residual off-policy RL framework that combines selective behavior regularization with value-guided action selection. Extensive experiments and analyses show that DICE-RL reliably improves performance with strong stability and sample efficiency. It enables mastery of complex long-horizon manipulation skills directly from high-dimensional pixel inputs, both in simulation and on a real robot. Project website: dice.rl.2026.

Robots are increasingly trained by watching people perform tasks, but this often leaves them with skills that look reasonable yet fail in small, important ways. This is especially problematic for long, precise tasks such as inserting a gear, screwing in a light bulb, or placing a belt around pulleys, where one small mistake can cause the whole attempt to fail. This paper presents DICE-RL, a way to turn an initially trained robot into a more reliable one using a limited amount of real practice. Instead of letting the robot explore freely, which can be slow, unsafe, or wasteful, DICE-RL keeps the robot close to behaviors it has already learned while using success-or-failure feedback to make small corrections. Over time, the robot becomes more likely to choose actions that lead to success and less likely to repeat actions that cause failures. We test this approach in simulated robot tasks and on a real robot. Across these settings, DICE-RL improves reliability and learns efficiently from relatively few practice attempts, helping robots master challenging manipulation skills from camera images.