Timezone: »
Oral
Model-based Reinforcement Learning for Continuous Control with Posterior Sampling
Ying Fan · Yifei Ming
Balancing exploration and exploitation is crucial in reinforcement learning (RL). In this paper, we study model-based posterior sampling for reinforcement learning (PSRL) in continuous state-action spaces theoretically and empirically. First, we show the first regret bound of PSRL in continuous spaces which is polynomial in the episode length to the best of our knowledge. With the assumption that reward and transition functions can be modeled by Bayesian linear regression, we develop a regret bound of $\tilde{O}(H^{3/2}d\sqrt{T})$, where $H$ is the episode length, $d$ is the dimension of the state-action space, and $T$ indicates the total time steps. This result matches the best-known regret bound of non-PSRL methods in linear MDPs. Our bound can be extended to nonlinear cases as well with feature embedding: using linear kernels on the feature representation $\phi$, the regret bound becomes $\tilde{O}(H^{3/2}d_{\phi}\sqrt{T})$, where $d_\phi$ is the dimension of the representation space. Moreover, we present MPC-PSRL, a model-based posterior sampling algorithm with model predictive control for action selection. To capture the uncertainty in models, we use Bayesian linear regression on the penultimate layer (the feature representation layer $\phi$) of neural networks. Empirical results show that our algorithm achieves the state-of-the-art sample efficiency in benchmark continuous control tasks compared to prior model-based algorithms, and matches the asymptotic performance of model-free algorithms.
Author Information
Ying Fan (University of Wisconsin-Madison)
Yifei Ming (University of Wisconsin-Madison)
Related Events (a corresponding poster, oral, or spotlight)
-
2021 Poster: Model-based Reinforcement Learning for Continuous Control with Posterior Sampling »
Wed. Jul 21st 04:00 -- 06:00 PM Room
More from the Same Authors
-
2022 : Are Vision Transformers Robust to Spurious Correlations ? »
Soumya Suvra Ghosal · Yifei Ming · Sharon Li -
2023 : Algorithms for Optimal Adaptation of Diffusion Models to Reward Functions »
Krishnamurthy Dvijotham · Shayegan Omidshafiei · Kimin Lee · Katie Collins · Deepak Ramachandran · Adrian Weller · Mohammad Ghavamzadeh · Milad Nasresfahani · Ying Fan · Jeremiah Liu -
2023 Poster: Optimizing DDPM Sampling with Shortcut Fine-Tuning »
Ying Fan · Kangwook Lee -
2022 Poster: Out-of-Distribution Detection with Deep Nearest Neighbors »
Yiyou Sun · Yifei Ming · Jerry Zhu · Sharon Li -
2022 Spotlight: Out-of-Distribution Detection with Deep Nearest Neighbors »
Yiyou Sun · Yifei Ming · Jerry Zhu · Sharon Li -
2022 Poster: POEM: Out-of-Distribution Detection with Posterior Sampling »
Yifei Ming · Ying Fan · Sharon Li -
2022 Oral: POEM: Out-of-Distribution Detection with Posterior Sampling »
Yifei Ming · Ying Fan · Sharon Li