Uncertainty-Guided Exploration and Stable Planning for Sparse-Reward Manipulation from Limited Demonstrations

Reinforcement learning from demonstrations (RLfD) offers a promising method for robotic manipulation with sparse rewards. However, limited demonstrations often cause agents to encounter out-of-distribution states where world models produce poor predictions. In multi-stage tasks, jointly optimizing a learned reward function and policy introduces a moving target problem, and the resulting non-stationarity intensifies the impact of uncertainty on policy learning. In this work, we propose QUEST, a model-based RL framework that adaptively switches between exploration and exploitation guided by uncertainty to achieve stable and efficient learning. Specifically, our approach employs intrinsic rewards to encourage exploration, leverages ensemble dynamics for uncertainty-guided planning, and introduces a hybrid sampling strategy to prioritize rare successful stage transitions. We evaluate QUEST on challenging sparse-reward manipulation tasks with limited expert demonstrations. Results show that QUEST outperforms state-of-the-art methods by 17\% on average, with gains increasing to 60\% on difficult tasks. We further demonstrate successful zero-shot sim-to-real transfer on five real-world tasks. Project website: https://quest-official.github.io/QUEST/.

Robots that can perform complex manipulation tasks, such as inserting a peg into a hole or stacking blocks, could one day help in factories, warehouses, and homes. Teaching robots such skills typically requires hundreds of human demonstrations, which is expensive and slow. We developed QUEST, a method that lets robots learn multi-step manipulation tasks from only 10 demonstrations. The key idea is that the robot continuously estimates how confident it is in its own predictions about the world. When the robot is uncertain, it explores carefully to gather more information. As its confidence grows, it plans more decisively to complete the task. This balance shifts automatically as learning progresses, so the robot neither stays stuck in cautious exploration nor takes reckless actions based on wrong assumptions. In simulation across 16 different manipulation tasks, QUEST achieved success rates 17% higher than existing methods on average, with improvements reaching 60% on the most difficult tasks. We also deployed it on a real robot performing five different tasks without any additional training in the real world, showing that the skills learned in simulation transfer reliably to the physical environment.