Timezone: »

 
Spotlight
On the Role of Discount Factor in Offline Reinforcement Learning
Hao Hu · yiqin yang · Qianchuan Zhao · Chongjie Zhang

Thu Jul 21 11:55 AM -- 12:00 PM (PDT) @ Room 301 - 303
in Reinforcement Learning »
Offline reinforcement learning (RL) enables effective learning from previously collected data without exploration, which shows great promise in real-world applications when exploration is expensive or even infeasible. The discount factor, $\gamma$, plays a vital role in improving online RL sample efficiency and estimation accuracy, but the role of the discount factor in offline RL is not well explored. This paper examines two distinct effects of $\gamma$ in offline RL with theoretical analysis, namely the regularization effect and the pessimism effect. On the one hand, $\gamma$ is a regulator to trade-off optimality with sample efficiency upon existing offline techniques. On the other hand, lower guidance $\gamma$ can also be seen as a way of pessimism where we optimize the policy's performance in the worst possible models. We empirically verify the above theoretical observation with tabular MDPs and standard D4RL tasks. The results show that the discount factor plays an essential role in the performance of offline RL algorithms, both under small data regimes upon existing offline methods and in large data regimes without other conservative methods.
Keywords: Reinforcement Learning · RL: Batch/Offline · RL: Discounted Cost/Reward · T: Reinforcement Learning and Planning

Author Information

Hao Hu (Tsinghua University)
yiqin yang (Tsinghua University)
Qianchuan Zhao (Tsinghua University)
Chongjie Zhang (Tsinghua University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors