Timezone: »

 
Poster
Towards Evaluating Adaptivity of Model-Based Reinforcement Learning Methods
Yi Wan · Ali Rahimi-Kalahroudi · Janarthanan Rajendran · Ida Momennejad · Sarath Chandar · Harm van Seijen

Wed Jul 20 03:30 PM -- 05:30 PM (PDT) @ Hall E #918
in Poster Session 2 »

In recent years, a growing number of deep model-based reinforcement learning (RL) methods have been introduced. The interest in deep model-based RL is not surprising, given its many potential benefits, such as higher sample efficiency and the potential for fast adaption to changes in the environment. However, we demonstrate, using an improved version of the recently introduced Local Change Adaptation (LoCA) setup, that well-known model-based methods such as PlaNet and DreamerV2 perform poorly in their ability to adapt to local environmental changes. Combined with prior work that made a similar observation about the other popular model-based method, MuZero, a trend appears to emerge, suggesting that current deep model-based methods have serious limitations. We dive deeper into the causes of this poor performance, by identifying elements that hurt adaptive behavior and linking these to underlying techniques frequently used in deep model-based RL. We empirically validate these insights in the case of linear function approximation by demonstrating that a modified version of linear Dyna achieves effective adaptation to local changes. Furthermore, we provide detailed insights into the challenges of building an adaptive nonlinear model-based method, by experimenting with a nonlinear version of Dyna.

Keywords: RL: Planning

Author Information

Yi Wan (University of Alberta)
Ali Rahimi-Kalahroudi (MILA - Université de Montréal)
Janarthanan Rajendran (Mila, University of Montreal)
Ida Momennejad (Microsoft Research)
Sarath Chandar (Mila / École Polytechnique de Montréal)
Harm van Seijen (Microsoft Research)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors