Timezone: »

 
Control-Oriented Model-Based Reinforcement Learning with Implicit Differentiation
Evgenii Nikishin · Romina Abachi · Rishabh Agarwal · Pierre-Luc Bacon

@
in ICML 2021 Workshop on Unsupervised Reinforcement Learning »

The shortcomings of maximum likelihood estimation in the context of model-based reinforcement learning have been highlighted by an increasing number of papers. When the model class is misspecified or has a limited representational capacity, model parameters with high likelihood might not necessarily result in high performance of the agent on a downstream control task. To alleviate this problem, we propose an end-to-end approach for model learning which directly optimizes the expected returns using implicit differentiation. We treat a value function that satisfies the Bellman optimality operator induced by the model as an implicit function of model parameters and show how to differentiate the function. We provide theoretical and empirical evidence highlighting the benefits of our approach in the model misspecification regime compared to likelihood-based methods.

Author Information

Evgenii Nikishin (Université de Montréal, MILA)
Romina Abachi (Department of Computer Science, University of Toronto)
Rishabh Agarwal (Google Research, Brain Team)
Pierre-Luc Bacon (University of Montreal)

More from the Same Authors