Timezone: »

 
Poster
Flow-based Recurrent Belief State Learning for POMDPs
Xiaoyu Chen · Yao Mu · Ping Luo · Shengbo Li · Jianyu Chen

Thu Jul 21 03:00 PM -- 05:00 PM (PDT) @ Hall E #1006
in Poster Session 3 »

Partially Observable Markov Decision Process (POMDP) provides a principled and generic framework to model real world sequential decision making processes but yet remains unsolved, especially for high dimensional continuous space and unknown models. The main challenge lies in how to accurately obtain the belief state, which is the probability distribution over the unobservable environment states given historical information. Accurately calculating this belief state is a precondition for obtaining an optimal policy of POMDPs. Recent advances in deep learning techniques show great potential to learn good belief states. However, existing methods can only learn approximated distribution with limited flexibility. In this paper, we introduce the \textbf{F}l\textbf{O}w-based \textbf{R}ecurrent \textbf{BE}lief \textbf{S}tate model (FORBES), which incorporates normalizing flows into the variational inference to learn general continuous belief states for POMDPs. Furthermore, we show that the learned belief states can be plugged into downstream RL algorithms to improve performance. In experiments, we show that our methods successfully capture the complex belief states that enable multi-modal predictions as well as high quality reconstructions, and results on challenging visual-motor control tasks show that our method achieves superior performance and sample efficiency.

Keywords: Reinforcement Learning · RL: Deep RL

Author Information

Xiaoyu Chen (Tsinghua University)
Yao Mu (The University of Hong Kong)

I am currently a Ph.D. Candidate of Computer Science at the University of Hong Kong, supervised by Prof. Ping Luo. Previously I obtained the M.Phil Degree under the supervision of Prof. Bo Cheng and Prof. Shengbo Li at the Intelligent Driving Laboratory from Tsinghua University in June 2021. Research Interests: Reinforcement Learning, Representation Learning, Autonomous Driving, and Computer Vision.

Ping Luo (The University of Hong Kong)
Shengbo Li (Tsinghua University)
Jianyu Chen (Tsinghua University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors