Skip to yearly menu bar Skip to main content


Calibrated Model-Based Deep Reinforcement Learning

Ali Malik · Volodymyr Kuleshov · Jiaming Song · Danny Nemer · Harlan Seymour · Stefano Ermon

Pacific Ballroom #36

Keywords: [ Planning and Control ] [ Deep Reinforcement Learning ] [ Bayesian Deep Learning ] [ Bandits ]


Estimates of predictive uncertainty are important for accurate model-based planning and reinforcement learning. However, predictive uncertainties --- especially ones derived from modern deep learning systems --- can be inaccurate and impose a bottleneck on performance. This paper explores which uncertainties are needed for model-based reinforcement learning and argues that ideal uncertainties should be calibrated, i.e. their probabilities should match empirical frequencies of predicted events. We describe a simple way to augment any model-based reinforcement learning agent with a calibrated model and show that doing so consistently improves planning, sample complexity, and exploration. On the \textsc{HalfCheetah} MuJoCo task, our system achieves state-of-the-art performance using 50\% fewer samples than the current leading approach. Our findings suggest that calibration can improve the performance of model-based reinforcement learning with minimal computational and implementation overhead.

Live content is unavailable. Log in and register to view live content