Invited Talk- Jesse Zhang: Beyond Human Feedback: Synthetic

Reinforcement learning has become a key ingredient in modern foundation models, driven by increasingly scalable sources of supervision ranging from human preferences to AI-generated feedback. Robotics, however, still relies heavily on hand-designed rewards or costly human annotations, limiting the scalability of reinforcement learning. In this talk, I will demonstrate how casting reward modeling as preference comparisons over robot behaviors enables scalable learning of reward functions for robotics. I will first show how zero-shot VLMs can act as reward judges by asking them to construct trajectory preferences instead of directly querying them for rewards, and then introduce Robometer, a learned reward model trained entirely from automatically generated preferences. Together, these methods demonstrate how synthetic feedback can transform reward engineering into a scalable learning problem, bringing reinforcement learning for robotics closer to the feedback-driven paradigm that has accelerated modern foundation models.