Timezone: »

 
Poster
GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values
Shangtong Zhang · Bo Liu · Shimon Whiteson

Tue Jul 14 08:00 AM -- 08:45 AM & Tue Jul 14 09:00 PM -- 09:45 PM (PDT) @ None #None
in AcceptedPapersParent »

We present GradientDICE for estimating the density ratio between the state distribution of the target policy and the sampling distribution in off-policy reinforcement learning. GradientDICE fixes several problems of GenDICE (Zhang et al., 2020), the current state-of-the-art for estimating such density ratios. Namely, the optimization problem in GenDICE is not a convex-concave saddle-point problem once nonlinearity in optimization variable parameterization is introduced to ensure positivity, so primal-dual algorithms are not guaranteed to find the desired solution. However, such nonlinearity is essential to ensure the consistency of GenDICE even with a tabular representation. This is a fundamental contradiction, resulting from GenDICE's original formulation of the optimization problem. In GradientDICE, we optimize a different objective from GenDICE by using the Perron-Frobenius theorem and eliminating GenDICE's use of divergence, such that nonlinearity in parameterization is not necessary for GradientDICE, which is provably convergent under linear function approximation.

Keywords: Deep Reinforcement Learning · Reinforcement Learning Theory · Reinforcement Learning - Theory

Author Information

Shangtong Zhang (University of Oxford)
Bo Liu (Auburn University)

Bo Liu is a tenure-track assistant professor in the Dept. of Computer Science and Software Engineering at Auburn University. He obtained his Ph.D. from Autonomous Learning Lab at University of Massachusetts Amherst, 2015, co-led by Drs. Sridhar Mahadevan and Andrew Barto. His primary research area covers decision-making under uncertainty, human-aided machine learning, symbolic AI, trustworthiness and interpretability in machine learning, and their numerous applications to BIGDATA, autonomous driving, and healthcare informatics. In his current research, he has more than 30 publications on several notable venues, such as NIPS, UAI, AAAI, IJCAI, AAMAS, JAIR, IEEE TNNLS, ACM TECS, etc. His research is funded by NSF, Amazon, Tencent (China), Adobe, and ETRI (South Korea). He is the recipient of the UAI'2015 Facebook best student paper award and the Amazon research award in 2018. His research results have been covered by many prestigious venues, including the classical textbook "Reinforcement Learning: An Introduction" (2nd edition), NIPS'2015/IJCAI'2016/AAAI'2019 tutorials.

Shimon Whiteson (University of Oxford)

More from the Same Authors