Timezone: »
The Laplacian representation recently gains increasing attention for reinforcement learning as it provides succinct and informative representation for states, by taking the eigenvectors of the Laplacian matrix of the state-transition graph as state embeddings. Such representation captures the geometry of the underlying state space and is beneficial to RL tasks such as option discovery and reward shaping. To approximate the Laplacian representation in large (or even continuous) state spaces, recent works propose to minimize a spectral graph drawing objective, which however has infinitely many global minimizers other than the eigenvectors. As a result, their learned Laplacian representation may differ from the ground truth. To solve this problem, we reformulate the graph drawing objective into a generalized form and derive a new learning objective, which is proved to have eigenvectors as its unique global minimizer. It enables learning high-quality Laplacian representations that faithfully approximate the ground truth. We validate this via comprehensive experiments on a set of gridworld and continuous control environments. Moreover, we show that our learned Laplacian representations lead to more exploratory options and better reward shaping.
Author Information
Kaixin Wang (National University of Singapore)
Kuangqi Zhou (National University of Singapore)
Qixin Zhang (city university of hong kong)
Jie Shao (Fudan University)
Bryan Hooi (National University of Singapore)
Jiashi Feng (National University of Singapore)
Related Events (a corresponding poster, oral, or spotlight)
-
2021 Poster: Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawing »
Wed. Jul 21st 04:00 -- 06:00 AM Room
More from the Same Authors
-
2022 Poster: Stochastic Continuous Submodular Maximization: Boosting via Non-oblivious Function »
Qixin Zhang · Zengde Deng · Zaiyi Chen · Haoyuan Hu · Yu Yang -
2022 Spotlight: Stochastic Continuous Submodular Maximization: Boosting via Non-oblivious Function »
Qixin Zhang · Zengde Deng · Zaiyi Chen · Haoyuan Hu · Yu Yang -
2022 Poster: The Geometry of Robust Value Functions »
Kaixin Wang · Navdeep Kumar · Kuangqi Zhou · Bryan Hooi · Jiashi Feng · Shie Mannor -
2022 Spotlight: The Geometry of Robust Value Functions »
Kaixin Wang · Navdeep Kumar · Kuangqi Zhou · Bryan Hooi · Jiashi Feng · Shie Mannor -
2021 Poster: CIFS: Improving Adversarial Robustness of CNNs via Channel-wise Importance-based Feature Selection »
Hanshu YAN · Jingfeng Zhang · Gang Niu · Jiashi Feng · Vincent Tan · Masashi Sugiyama -
2021 Spotlight: CIFS: Improving Adversarial Robustness of CNNs via Channel-wise Importance-based Feature Selection »
Hanshu YAN · Jingfeng Zhang · Gang Niu · Jiashi Feng · Vincent Tan · Masashi Sugiyama -
2020 Poster: Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation »
Jian Liang · Dapeng Hu · Jiashi Feng -
2018 Poster: Policy Optimization with Demonstrations »
Bingyi Kang · Zequn Jie · Jiashi Feng -
2018 Poster: WSNet: Compact and Efficient Networks Through Weight Sampling »
Xiaojie Jin · Yingzhen Yang · Ning Xu · Jianchao Yang · Nebojsa Jojic · Jiashi Feng · Shuicheng Yan -
2018 Oral: WSNet: Compact and Efficient Networks Through Weight Sampling »
Xiaojie Jin · Yingzhen Yang · Ning Xu · Jianchao Yang · Nebojsa Jojic · Jiashi Feng · Shuicheng Yan -
2018 Oral: Policy Optimization with Demonstrations »
Bingyi Kang · Zequn Jie · Jiashi Feng -
2018 Poster: Understanding Generalization and Optimization Performance of Deep CNNs »
Pan Zhou · Jiashi Feng -
2018 Oral: Understanding Generalization and Optimization Performance of Deep CNNs »
Pan Zhou · Jiashi Feng