Timezone: »
Offline Reinforcement Learning promises to learn effective policies from previously-collected, static datasets without the need for exploration. However, existing Q-learning and actor-critic based off-policy RL algorithms fail when bootstrapping from out-of-distribution (OOD) actions or states. We hypothesize that a key missing ingredient from the existing methods is a proper treatment of uncertainty in the offline setting. We propose Uncertainty Weighted Actor-Critic (UWAC), an algorithm that detects OOD state-action pairs and down-weights their contribution in the training objectives accordingly. Implementation-wise, we adopt a practical and effective dropout-based uncertainty estimation method that introduces very little overhead over existing RL algorithms. Empirically, we observe that UWAC substantially improves model stability during training. In addition, UWAC out-performs existing offline RL methods on a variety of competitive tasks, and achieves significant performance gains over the state-of-the-art baseline on datasets with sparse demonstrations collected from human experts.
Author Information
Yue Wu (Carnegie Mellon University)
Shuangfei Zhai (Apple)
Nitish Srivastava (Apple)
Joshua M Susskind (Apple, Inc.)
Jian Zhang (Apple Inc.)
AI and Robotics. AI Research & Autonomous System Technologies at Apple
Ruslan Salakhutdinov (Carnegie Mellen University)
Hanlin Goh (Apple)
Related Events (a corresponding poster, oral, or spotlight)
-
2021 Poster: Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning »
Wed. Jul 21st 04:00 -- 06:00 AM Room
More from the Same Authors
-
2021 : Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks »
Etai Littwin · Omid Saremi · Shuangfei Zhai · Vimal Thilak · Hanlin Goh · Joshua M Susskind · Greg Yang -
2021 : Implicit Greedy Rank Learning in Autoencoders via Overparameterized Linear Networks »
Shih-Yu Sun · Vimal Thilak · Etai Littwin · Omid Saremi · Joshua M Susskind -
2021 : Online Sub-Sampling for Reinforcement Learning with General Function Approximation »
Dingwen Kong · Ruslan Salakhutdinov · Ruosong Wang · Lin Yang -
2023 : Plan, Eliminate, and Track --- Language Models are Good Teachers for Embodied Agents. »
Yue Wu · So Yeon Min · Yonatan Bisk · Ruslan Salakhutdinov · Amos Azaria · Yuanzhi Li · Tom Mitchell · Shrimai Prabhumoye -
2023 : SPRING: Studying Papers and Reasoning to play Games »
Yue Wu · Shrimai Prabhumoye · So Yeon Min · Yonatan Bisk · Ruslan Salakhutdinov · Amos Azaria · Tom Mitchell · Yuanzhi Li -
2023 : BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping »
Jiatao Gu · Shuangfei Zhai · Yizhe Zhang · Lingjie Liu · Joshua M Susskind -
2023 Poster: Graph Generative Model for Benchmarking Graph Neural Networks »
Minji Yoon · Yue Wu · John Palowitch · Bryan Perozzi · Ruslan Salakhutdinov -
2023 Poster: Stabilizing Transformer Training by Preventing Attention Entropy Collapse »
Shuangfei Zhai · Tatiana Likhomanenko · Etai Littwin · Dan Busbridge · Jason Ramapuram · Yizhe Zhang · Jiatao Gu · Joshua M Susskind -
2023 Poster: NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion »
Jiatao Gu · Alex Trevithick · Kai-En Lin · Joshua M Susskind · Christian Theobalt · Lingjie Liu · Ravi Ramamoorthi -
2022 Poster: Efficient Representation Learning via Adaptive Context Pooling »
Chen Huang · Walter Talbott · Navdeep Jaitly · Joshua M Susskind -
2022 Spotlight: Efficient Representation Learning via Adaptive Context Pooling »
Chen Huang · Walter Talbott · Navdeep Jaitly · Joshua M Susskind -
2022 Poster: Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs »
Tianwei Ni · Benjamin Eysenbach · Ruslan Salakhutdinov -
2022 Spotlight: Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs »
Tianwei Ni · Benjamin Eysenbach · Ruslan Salakhutdinov -
2022 Poster: Position Prediction as an Effective Pretraining Strategy »
Shuangfei Zhai · Navdeep Jaitly · Jason Ramapuram · Dan Busbridge · Tatiana Likhomanenko · Joseph Cheng · Walter Talbott · Chen Huang · Hanlin Goh · Joshua M Susskind -
2022 Spotlight: Position Prediction as an Effective Pretraining Strategy »
Shuangfei Zhai · Navdeep Jaitly · Jason Ramapuram · Dan Busbridge · Tatiana Likhomanenko · Joseph Cheng · Walter Talbott · Chen Huang · Hanlin Goh · Joshua M Susskind -
2021 Poster: Towards Understanding and Mitigating Social Biases in Language Models »
Paul Liang · Chiyu Wu · LP Morency · Ruslan Salakhutdinov -
2021 Poster: Reasoning Over Virtual Knowledge Bases With Open Predicate Relations »
Haitian Sun · Patrick Verga · Bhuwan Dhingra · Ruslan Salakhutdinov · William Cohen -
2021 Spotlight: Reasoning Over Virtual Knowledge Bases With Open Predicate Relations »
Haitian Sun · Patrick Verga · Bhuwan Dhingra · Ruslan Salakhutdinov · William Cohen -
2021 Spotlight: Towards Understanding and Mitigating Social Biases in Language Models »
Paul Liang · Chiyu Wu · LP Morency · Ruslan Salakhutdinov -
2021 Poster: Instabilities of Offline RL with Pre-Trained Neural Representation »
Ruosong Wang · Yifan Wu · Ruslan Salakhutdinov · Sham Kakade -
2021 Spotlight: Instabilities of Offline RL with Pre-Trained Neural Representation »
Ruosong Wang · Yifan Wu · Ruslan Salakhutdinov · Sham Kakade -
2021 Poster: Information Obfuscation of Graph Neural Networks »
Peiyuan Liao · Han Zhao · Keyulu Xu · Tommi Jaakkola · Geoff Gordon · Stefanie Jegelka · Ruslan Salakhutdinov -
2021 Poster: On Proximal Policy Optimization's Heavy-tailed Gradients »
Saurabh Garg · Joshua Zhanson · Emilio Parisotto · Adarsh Prasad · Zico Kolter · Zachary Lipton · Sivaraman Balakrishnan · Ruslan Salakhutdinov · Pradeep Ravikumar -
2021 Spotlight: On Proximal Policy Optimization's Heavy-tailed Gradients »
Saurabh Garg · Joshua Zhanson · Emilio Parisotto · Adarsh Prasad · Zico Kolter · Zachary Lipton · Sivaraman Balakrishnan · Ruslan Salakhutdinov · Pradeep Ravikumar -
2021 Spotlight: Information Obfuscation of Graph Neural Networks »
Peiyuan Liao · Han Zhao · Keyulu Xu · Tommi Jaakkola · Geoff Gordon · Stefanie Jegelka · Ruslan Salakhutdinov -
2020 Workshop: Workshop on Learning in Artificial Open Worlds »
Arthur Szlam · Katja Hofmann · Ruslan Salakhutdinov · Noboru Kuno · William Guss · Kavya Srinet · Brandon Houghton -
2020 Workshop: Bridge Between Perception and Reasoning: Graph Neural Networks & Beyond »
Jian Tang · Le Song · Jure Leskovec · Renjie Liao · Yujia Li · Sanja Fidler · Richard Zemel · Ruslan Salakhutdinov -
2020 Poster: Equivariant Neural Rendering »
Emilien Dupont · Miguel Angel Bautista Martin · Alex Colburn · Aditya Sankar · Joshua M Susskind · Qi Shan -
2019 Poster: Addressing the Loss-Metric Mismatch with Adaptive Loss Alignment »
Chen Huang · Shuangfei Zhai · Walter Talbott · Miguel Angel Bautista Martin · Shih-Yu Sun · Carlos Guestrin · Joshua M Susskind -
2019 Oral: Addressing the Loss-Metric Mismatch with Adaptive Loss Alignment »
Chen Huang · Shuangfei Zhai · Walter Talbott · Miguel Angel Bautista Martin · Shih-Yu Sun · Carlos Guestrin · Joshua M Susskind -
2019 Talk: Opening Remarks »
Kamalika Chaudhuri · Ruslan Salakhutdinov -
2018 Poster: Transformation Autoregressive Networks »
Junier Oliva · Kumar Avinava Dubey · Manzil Zaheer · Barnabás Póczos · Ruslan Salakhutdinov · Eric Xing · Jeff Schneider -
2018 Oral: Transformation Autoregressive Networks »
Junier Oliva · Kumar Avinava Dubey · Manzil Zaheer · Barnabás Póczos · Ruslan Salakhutdinov · Eric Xing · Jeff Schneider -
2018 Poster: Structured Control Nets for Deep Reinforcement Learning »
Mario Srouji · Jian Zhang · Ruslan Salakhutdinov -
2018 Poster: Gated Path Planning Networks »
Lisa Lee · Emilio Parisotto · Devendra Singh Chaplot · Eric Xing · Ruslan Salakhutdinov -
2018 Oral: Structured Control Nets for Deep Reinforcement Learning »
Mario Srouji · Jian Zhang · Ruslan Salakhutdinov -
2018 Oral: Gated Path Planning Networks »
Lisa Lee · Emilio Parisotto · Devendra Singh Chaplot · Eric Xing · Ruslan Salakhutdinov -
2017 Poster: Toward Controlled Generation of Text »
Zhiting Hu · Zichao Yang · Xiaodan Liang · Ruslan Salakhutdinov · Eric Xing -
2017 Poster: Improved Variational Autoencoders for Text Modeling using Dilated Convolutions »
Zichao Yang · Zhiting Hu · Ruslan Salakhutdinov · Taylor Berg-Kirkpatrick -
2017 Talk: Improved Variational Autoencoders for Text Modeling using Dilated Convolutions »
Zichao Yang · Zhiting Hu · Ruslan Salakhutdinov · Taylor Berg-Kirkpatrick -
2017 Talk: Toward Controlled Generation of Text »
Zhiting Hu · Zichao Yang · Xiaodan Liang · Ruslan Salakhutdinov · Eric Xing