Timezone: »
Exogenous state variables and rewards can slow down reinforcement learning by injecting uncontrolled variation into the reward signal. We formalize exogenous state variables and rewards and identify conditions under which an MDP with exogenous state can be decomposed into an exogenous Markov Reward Process involving only the exogenous state+reward and an endogenous Markov Decision Process defined with respect to only the endogenous rewards. We also derive a variance-covariance condition under which Monte Carlo policy evaluation on the endogenous MDP is accelerated compared to using the full MDP. Similar speedups are likely to carry over to all RL algorithms. We develop two algorithms for discovering the exogenous variables and test them on several MDPs. Results show that the algorithms are practical and can significantly speed up reinforcement learning.
Author Information
Thomas Dietterich ((organization))
George Trimponias (Huawei Noah's Ark Lab)
Zhitang Chen (Huawei Noah’s Ark Lab)
Related Events (a corresponding poster, oral, or spotlight)
-
2018 Poster: Discovering and Removing Exogenous State Variables and Rewards for Reinforcement Learning »
Thu. Jul 12th 04:15 -- 07:00 PM Room Hall B #17
More from the Same Authors
-
2023 Poster: Weakly Supervised Disentangled Generative Causal Representation Learning »
Xinwei Shen · Furui Liu · Hanze Dong · Qing Lian · Zhitang Chen · Tong Zhang -
2021 : Live Panel Discussion »
Thomas Dietterich · Chelsea Finn · Kamalika Chaudhuri · Yarin Gal · Uri Shalit -
2021 : RL Foundation Panel »
Matthew Botvinick · Thomas Dietterich · Leslie Kaelbling · John Langford · Warrren B Powell · Csaba Szepesvari · Lihong Li · Yuxi Li -
2019 : Panel Discussion (moderator: Tom Dietterich) »
Max Welling · Kilian Weinberger · Terrance Boult · Dawn Song · Thomas Dietterich -
2018 Poster: Open Category Detection with PAC Guarantees »
Si Liu · Risheek Garrepalli · Thomas Dietterich · Alan Fern · Dan Hendrycks -
2018 Oral: Open Category Detection with PAC Guarantees »
Si Liu · Risheek Garrepalli · Thomas Dietterich · Alan Fern · Dan Hendrycks