Timezone: »

 
Oral
Adversarially Trained Actor Critic for Offline Reinforcement Learning
Ching-An Cheng · Tengyang Xie · Nan Jiang · Alekh Agarwal

Thu Jul 21 11:15 AM -- 11:35 AM (PDT) @ Room 301 - 303
in Reinforcement Learning »

We propose Adversarially Trained Actor Critic (ATAC), a new model-free algorithm for offline reinforcement learning (RL) under insufficient data coverage, based on the concept of relative pessimism. ATAC is designed as a two-player Stackelberg game framing of offline RL: A policy actor competes against an adversarially trained value critic, who finds data-consistent scenarios where the actor is inferior to the data-collection behavior policy. We prove that, when the actor attains no regret in the two-player game, running ATAC produces a policy that provably 1) outperforms the behavior policy over a wide range of hyperparameters that control the degree of pessimism, and 2) competes with the best policy covered by data with appropriately chosen hyperparameters. Compared with existing works, notably our framework offers both theoretical guarantees for general function approximation and a deep RL implementation scalable to complex environments and large datasets. In the D4RL benchmark, ATAC consistently outperforms state-of-the-art offline RL algorithms on a range of continuous control tasks.

Keywords: RL: Batch/Offline · RL: Function Approximation · RL: Policy Search · T: Reinforcement Learning and Planning

Author Information

Ching-An Cheng (Microsoft Research)
Tengyang Xie (University of Illinois at Urbana-Champaign)
Nan Jiang (University of Illinois at Urbana-Champaign)
Alekh Agarwal (Microsoft Research)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors