CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning

Reward design remains a central challenge in reinforcement learning (RL). Hand-crafted rewards are often difficult to specify and may lead to suboptimal policies, while learned rewards from preferences can suffer from inefficiency and unstable training. Inspired by the dual nature of human learning explored in cognitive science, we decompose rewards into two complementary components: Formal Rewards (FR), explicitly designed based on task knowledge, and Residual Rewards (RR), learned from observations to capture implicit and nuanced preferences. Based on this decomposition, we propose CoRe, a hybrid framework that integrates FR and RR with vision-language models (VLMs) feedback to achieve preference-aligned policies without human involvement. Our contributions are twofold: (1) We propose a Formal Reward Module (FRM) that leverages VLMs to iteratively design and optimize FR based on task knowledge and preference feedback, enabling the continual improvement of policy during training; (2) We introduce a Residual Reward Module (RRM) that learns RR from video-level preference by employing VLMs to generate preference labels and capturing nuanced rewards that complement FR, ensuring alignment with human intent. Through the synergy of FRM and RRM, CoRe enables the automatic construction of reliable rewards that are efficient and preference-aligned. Extensive experiments demonstrate that CoRe outperforms existing approaches in terms of policy learning effectiveness and efficiency on ten robotic manipulation tasks in simulation and five real-worlds.

Robots can learn new skills through trial and error, but teaching them what counts as “good behavior” is often difficult. In many cases, researchers must carefully design reward signals by hand, which is time-consuming and may lead robots to exploit shortcuts that do not match human expectations. Existing methods that learn rewards from human feedback also tend to require large amounts of labeled data and unstable training. In this work, we introduce CoRe, a framework that helps robots learn behaviors that better match human intentions while reducing the need for manual reward design. Inspired by how humans combine prior knowledge with experience, our method splits rewards into two parts: one captures clear task goals that can be explicitly described, while the other learns subtle preferences from visual observations. We use large vision-language models to automatically provide feedback by comparing videos of robot behaviors and identifying which actions better satisfy the task. We tested CoRe on a wide range of robotic manipulation tasks, including opening drawers, inserting objects, folding cloth, and straightening ropes, both in simulation and on real robots. Our method learned faster, required much less feedback than previous approaches, and produced more reliable and human-aligned robot behaviors.