Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates

Inverse reinforcement learning (IRL) is typically formulated as maximizing entropy subject to matching the distribution of expert trajectories. Classical (dual-ascent) IRL guarantees monotonic performance improvement but requires fully solving an RL problem each iteration to compute dual gradients. More recent adversarial methods avoid this cost at the expense of stability and monotonic dual improvement, by directly optimizing the primal problem and using a discriminator to provide rewards. In this work, we bridge the gap between these approaches by enabling monotonic improvement of the reward function and policy without having to fully solve an RL problem at every iteration. Our key theoretical insight is that a trust-region-optimal policy for a reward function update can be globally optimal for a smaller update in the same direction. This smaller update allows us to explicitly optimize the dual objective while only relying on a local search around the current policy. In doing so, our approach avoids the training instabilities of adversarial methods, offers monotonic performance improvement, and learns a reward function in the traditional sense of IRL—one that can be globally optimized to match expert demonstrations. Our proposed algorithm, Trust Region Inverse Reinforcement Learning (TRIRL), outperforms state-of-the-art imitation learning methods across multiple challenging tasks by a factor of 2.4x in terms of aggregate inter-quartile mean, while recovering reward functions that generalize to system dynamics shifts.

Teaching robots to imitate human demonstrations usually involves understanding what goal the human was pursuing—a "reward function"—and then training the robot to pursue that goal. Learning rewards rather than just copying movements matters because a robot that understands why an expert acts a certain way can adapt its behavior to new situations, rather than blindly reproducing motions that break down when conditions change. The classical method for learning rewards is reliable but prohibitively slow in the real world: each time the reward estimate is refined, the robot must be retrained to act as per this reward. Modern methods skip this retraining using adversarial learning, but the resulting method is unstable, hard to tune, and the recovered reward only makes sense in the specific environment where it was learned. This paper provides a useful mathematical shortcut. Instead of fully training the robot on a new reward, we instead only train it until a limit (called "trust region"). We show that this limited training still results in optimal behaviour if the reward function is also mathematically corrected to account for this partial optimization. This avoids both the expensive, full retraining of classical methods and the instability of the adversarial alternative. The resulting algorithm trains more reliably, performs 2.4 times better than competing methods, and recovers rewards that remain meaningful even when the robot's environment changes.