MAVRL: Learning Reward Functions from Multiple Feedback Types with Amortized Variational Inference

Reward learning typically relies on a single feedback type or combines multiple feedback types using manually weighted loss terms. Currently, it remains unclear how to jointly learn reward functions from heterogeneous feedback types such as demonstrations, comparisons, ratings, rankings, and stops that provide qualitatively different signals. We address this challenge by formulating reward learning from multiple feedback types as Bayesian inference over a shared latent reward function, where each feedback type contributes information through an explicit likelihood. We introduce a scalable amortized variational inference approach that learns a shared reward encoder and feedback-specific likelihood decoders and is trained by optimizing a single evidence lower bound. Our approach avoids reducing feedback to a common intermediate representation and eliminates the need for manual loss balancing. Across discrete and continuous-control benchmarks, we show that jointly inferred reward posteriors outperform single-type baselines, exploit complementary information across feedback types, and yield policies that are more robust to environment perturbations. The inferred reward uncertainty further provides interpretable signals for analyzing model confidence and consistency across feedback types.

Teaching an AI system what we actually want it to do is surprisingly hard. Writing down a precise rulebook for "good behavior" almost always misses important details, so researchers instead let the AI learn from human feedback. But people give feedback in many different ways: by demonstrating how to do something, by picking which of two attempts they preferred, by giving a star rating, or by hitting a stop button when the AI is doing something unsafe. Each of these signals tells the AI something different about what we want, and on its own each leaves large gaps. Today, most methods either focus on a single feedback type or hand-tune how much weight to give each one, which is brittle and often arbitrary. We introduce MAVRL, a method that learns from all of these feedback types at the same time. Instead of picking one or manually balancing them, MAVRL treats each kind of feedback as a noisy clue about the same underlying notion of "what the human wants" and combines them using standard tools from probability theory. A single shared model then figures out for itself how much each feedback type should count. Across a range of test environments, combining feedback types reliably beat using any single type alone. The combined model also recovered the intended goal more accurately, produced behavior that held up better when the environment was perturbed, and reported its own uncertainty in a way that flags where feedback was scarce or contradictory.