Timezone: »

 
Poster
Calibrated Model-Based Deep Reinforcement Learning
Ali Malik · Volodymyr Kuleshov · Jiaming Song · Danny Nemer · Harlan Seymour · Stefano Ermon

Thu Jun 13 06:30 PM -- 09:00 PM (PDT) @ Pacific Ballroom #36
in Posters Thu »

Estimates of predictive uncertainty are important for accurate model-based planning and reinforcement learning. However, predictive uncertainties --- especially ones derived from modern deep learning systems --- can be inaccurate and impose a bottleneck on performance. This paper explores which uncertainties are needed for model-based reinforcement learning and argues that ideal uncertainties should be calibrated, i.e. their probabilities should match empirical frequencies of predicted events. We describe a simple way to augment any model-based reinforcement learning agent with a calibrated model and show that doing so consistently improves planning, sample complexity, and exploration. On the \textsc{HalfCheetah} MuJoCo task, our system achieves state-of-the-art performance using 50\% fewer samples than the current leading approach. Our findings suggest that calibration can improve the performance of model-based reinforcement learning with minimal computational and implementation overhead.

Keywords: Bandits · Bayesian Deep Learning · Deep Reinforcement Learning · Planning and Control

Author Information

Ali Malik (Stanford Universtiy)
Volodymyr Kuleshov (Stanford University / Afresh)
Jiaming Song (Stanford)
Danny Nemer (Afresh Technologies)
Harlan Seymour (Afresh Technologies)
Stefano Ermon (Stanford University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors