Timezone: »

Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
Shentao Yang · Yihao Feng · Shujian Zhang · Mingyuan Zhou

Tue Jul 19 11:55 AM -- 12:00 PM (PDT) @ Room 309

Offline reinforcement learning (RL) extends the paradigm of classical RL algorithms to purely learning from static datasets, without interacting with the underlying environment during the learning process. A key challenge of offline RL is the instability of policy training, caused by the mismatching between the distribution of the offline data and the undiscounted stationary state-action distribution of the learned policy. To avoid the detrimental impact of distribution mismatching, we regularize the undiscounted stationary distribution of the current policy towards the offline data during the policy improvement. Further, we train a dynamics model for regularization and to better estimate the stationary distribution of the current policy, reducing the error induced by distribution mismatch. On a wide range of continuous-control offline datasets, our method indicates competitive performance, which validates our algorithm.

Author Information

Shentao Yang (University of Texas at Austin)
Yihao Feng (The University of Texas at Austin)
Shujian Zhang (UT Austin)
Mingyuan Zhou (University of Texas at Austin)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors