Timezone: »

 
Poster
Computationally Efficient PAC RL in POMDPs with Latent Determinism and Conditional Embeddings
Masatoshi Uehara · Ayush Sekhari · Jason Lee · Nathan Kallus · Wen Sun

Wed Jul 26 02:00 PM -- 03:30 PM (PDT) @ Exhibit Hall 1 #808
in Poster Session 3 »
We study reinforcement learning with function approximation for large-scale Partially Observable Markov Decision Processes (POMDPs) where the state space and observation space are large or even continuous. Particularly, we consider Hilbert space embeddings of POMDP where the feature of latent states and the feature of observations admit a conditional Hilbert space embedding of the observation emission process, and the latent state transition is deterministic. Under the function approximation setup where the optimal latent state-action $Q$-function is linear in the state feature, and the optimal $Q$-function has a gap in actions, we provide a computationally and statistically efficient algorithm for finding the exact optimal policy. We show our algorithm's computational and statistical complexities scale polynomially with respect to the horizon and the intrinsic dimension of the feature on the observation space. Furthermore, we show both the deterministic latent transitions and gap assumptions are necessary to avoid statistical complexity exponential in horizon or dimension. Since our guarantee does not have an explicit dependence on the size of the state and observation spaces, our algorithm provably scales to large-scale POMDPs.
[ PDF] 

Author Information

Masatoshi Uehara (Cornell University)
Ayush Sekhari (Cornell University)
Jason Lee (Princeton University)
Nathan Kallus (Cornell University)
Wen Sun (Cornell University)

More from the Same Authors