ICML 2020

Timezone: »

Workshop

Inductive Biases, Invariances and Generalization in Reinforcement Learning

Anirudh Goyal · Rosemary Nan Ke · Stefan Bauer · Jane Wang · Theophane Weber · Fabio Viola · Bernhard Schölkopf · Stefan Bauer

Sat Jul 18 03:00 AM -- 03:00 AM (PDT) @ None

in Workshops Sat »

Event URL: https://biases-invariances-generalization.github.io/ »

One proposed solution towards the goal of designing machines that can extrapolate experience across environments and tasks, are inductive biases. Providing and starting algorithms with inductive biases might help to learn invariances e.g. a causal graph structure, which in turn will allow the agent to generalize across environments and tasks.
While some inductive biases are already available and correspond to common knowledge, one key requirement to learn inductive biases from data seems to be the possibility to perform and learn from interventions. This assumption is partially motivated by the accepted hypothesis in psychology about the need to experiment in order to discover causal relationships. This corresponds to an reinforcement learning environment, where the agent can discover causal factors through interventions and observing their effects.

We believe that one reason which has hampered progress on building intelligent agents is the limited availability of good inductive biases. Learning inductive biases from data is difficult since this corresponds to an interactive learning setting, which compared to classical regression or classification frameworks is far less understood e.g. even formal definitions of generalization in RL have not been developed. While Reinforcement Learning has already achieved impressive results, the sample complexity required to achieve consistently good performance is often prohibitively high. This has limited most RL to either games or settings where an accurate simulator is available. Another issue is that RL agents are often brittle in the face of even tiny changes to the environment (either visual or mechanistic changes) unseen in the training phase.

To build intuition for the scope of the generalization problem in RL, consider the task of training a robotic car mechanic that can diagnose and repair any problem with a car. Current methods are all insufficient in some respect -- on-policy policy gradient algorithms need to cycle through all possible broken cars on every single iteration, off-policy algorithms end up with a mess of instability due to perception and highly diverse data, and model-based methods may struggle to fully estimate a complex web of causality.

In our workshop we hope to explore research and new ideas on topics related to inductive biases, invariances and generalization, including:

- What are efficient ways to learn inductive biases from data?
- Which inductive biases are most suitable to achieve generalization?
- Can we make the problem of generalization in particular for RL more concrete and figure out standard terms for discussing the problem?
- Causality and generalization especially in RL
- Model-based RL and generalization.
- Sample Complexity in reinforcement learning.
- Can we create models that are robust visual environments, assuming all the underlying mechanics are the same. Should this count as generalization or transfer learning?
- Robustness to changes in the mechanics of the environment, such as scaling of rewards.
- Can we create a theoretical understanding of generalization in RL, and understand how it is related to the well developed ideas from statistical learning theory.
- in RL, the training data is collected by the agent and it is affected by the agent's policy.
Therefore, the training distribution is not a fixed distribution. How does this affect how we should think about generalization?

The question of generalization in reinforcement learning is essential to the field’s future both in theory and in practice. However there are still open questions about the right way to think about generalization in RL, the right way to formalize the problem, and the most important tasks. This workshop would help to address this issue by bringing together researchers from different backgrounds to discuss these challenges.

Sat 3:00 a.m. - 3:15 a.m.	Opening remarks
Sat 3:15 a.m. - 4:30 a.m.	Poster Session 1 (Poster Session)
Sat 4:30 a.m. - 5:00 a.m.	Invited talk 1 Silver (Talk) » Video Meta Gradient Reinforcement Learning Link »	David Silver
Sat 5:00 a.m. - 5:10 a.m.	QA for invited talk 1 Silver (10min QA)	David Silver
Sat 5:10 a.m. - 5:40 a.m.	Invited talk 2 Uhler (Talk) » Video Multi-Domain Data Integration: From Observations to Mechanistic Insights Link »	Caroline Uhler
Sat 5:40 a.m. - 5:50 a.m.	QA for invited talk 2 Uhler (10min QA)	Caroline Uhler
Sat 5:50 a.m. - 6:05 a.m.	Automatic Data Augmentation for Generalization in Reinforcement Learning (Spotlight) » Video http://slideslive.com/38931360 Link »	Roberta Raileanu
Sat 6:15 a.m. - 7:30 a.m.	Poster Session 2 (Poster Session)
Sat 7:30 a.m. - 8:10 a.m.	Invited talk 3 Yang (Talk) » Video Augmenting data to improve robustness – a blessing or a curse? Link »	Fanny Yang
Sat 8:10 a.m. - 8:40 a.m.	Invited talk 4 Bengio (Talk) » Video System 2 Priors Link »	Yoshua Bengio
Sat 8:40 a.m. - 8:50 a.m.	QA for invited talk 4 Bengio (10min QA)	Yoshua Bengio
Sat 8:50 a.m. - 9:05 a.m.	Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers (Spotlight) » Video http://slideslive.com/38931338 Link »	Swapnil Asawa, Benjamin Eysenbach
Sat 9:05 a.m. - 9:15 a.m.	QA for invited talk 3 Yang (10min QA)	Fanny Yang
Sat 9:15 a.m. - 9:45 a.m.	Invited talk 5 White (Talk) » Video A New RNN algorithm using the computational inductive bias of span independence Link »	Martha White
Sat 9:45 a.m. - 9:55 a.m.	QA for invited talk 5 White (10min QA)	Martha White
Sat 10:00 a.m. - 10:30 a.m.	Invited talk 6 Heess (Talk) » Video From skills to tasks: Reusing and generalizing knowledge for motor control Link »	Nicolas Heess
Sat 10:30 a.m. - 10:40 a.m.	QA for invited talk 6 Heess (10min QA)	Nicolas Heess
Sat 10:40 a.m. - 11:55 a.m.	Poster Session 3 (Poster Session)
Sat 11:55 a.m. - 12:25 p.m.	Invited talk 7 Wang (Talk) » Video Statistical Complexity of RL and the use of regression Link »	Mengdi Wang
Sat 12:25 p.m. - 12:35 p.m.	QA for invited talk 7 Wang (10min QA)	Mengdi Wang
Sat 12:35 p.m. - 1:05 p.m.	Invited talk 8 Kakade (Talk) » Video On the Theory of Policy Gradient Methods: Optimality, Generalization and Distribution Shift	Sham Kakade
Sat 1:05 p.m. - 1:15 p.m.	QA for invited talk 8 Kakade (10min QA)	Sham Kakade
Sat 1:15 p.m. - 2:15 p.m.	Panel Discussion Session (Discussion Panel)
-	Evaluating Agents without Rewards (Poster) » Video http://slideslive.com/38931301 Link »	Danijar Hafner
-	Reinforcement Learning Generalization with Surprise Minimization (Poster) » Video http://slideslive.com/38931304 Link »	Jerry Zikun Chen
-	Learning Action Priors for Visuomotor transfer (Poster) » Video http://slideslive.com/38931326 Link »	aajay3110 Ajay
-	MOPO: Model-based Offline Policy Optimization (Poster) » Video http://slideslive.com/38931302 Link »	Tianhe Yu
-	Meta Attention Networks: Meta Learning Attention To Modulate Information Between Sparsely Interacting Recurrent Modules (Poster) » Video http://slideslive.com/38931327 Link »	Kanika Madan
-	Spatially Structured Recurrent Modules (Poster) » Video http://slideslive.com/38931303 Link »	Nasim Rahaman
-	Neural Dynamic Policies for End-to-End Sensorimotor Learning (Poster) » Video http://slideslive.com/38931329 Link »	Abhinav Gupta
-	Watch your Weight Reinforcement Learning (Poster) » Video http://slideslive.com/38931330 Link »	Robert Müller
-	HAT: Hierarchical Alternative Training for Long Range Policy Transfer (Poster) » Video http://slideslive.com/38931331 Link »	Min Sun, Wei-Cheng Tseng
-	Learning to Learn from Failures Using Replay (Poster) » Video http://slideslive.com/38931328 Link »	Tao Chen
-	Meta-Reinforcement Learning for Robotic Industrial Insertion Tasks (Poster) » Video http://slideslive.com/38931362 Link »	Gerrit Schoettler
-	Bridging Worlds in Reinforcement Learning with Model-Advantage (Poster) » Video http://slideslive.com/38931333 Link »	Ashwin Kalyan, Nirbhay Modhe
-	Fighting Copycat Agents in Behavioral Cloning From Multiple Observations (Poster) » Video http://slideslive.com/38931530 Link »	Chuan Wen
-	Conditioning of Reinforcement Learning Agents and its Policy Regularization Application (Poster (5 min)) » Video http://slideslive.com/38931436 Link »	Arip Asadulaev
-	Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors (Poster) » Video http://slideslive.com/38931332 Link »	Karl Pertsch
-	Planning to Explore via Self-Supervised World Models (Poster) » Video http://slideslive.com/38931334 Link »	Ramanan Sekar
-	Robust Reinforcement Learning using Adversarial Populations (Poster) » Video http://slideslive.com/38931336 Link »	Eugene Vinitsky
-	Structure Mapping for Transferability of Causal Models (Poster) » Video http://slideslive.com/38931337 Link »	Purva Pruthi
-	Efficient Imitation Learning with Local Trajectory Optimization (Poster) » Video http://slideslive.com/38931339 Link »	Jialin Song
-	Efficient Adaptation for End-to-End Vision-Based Robotic Manipulation (Poster) » Video http://slideslive.com/38931340 Link »	Gaurav Sukhatme
-	Exact (Then Approximate) Dynamic Programming for Deep Reinforcement Learning (Poster) » Video http://slideslive.com/38931341 Link »	Henrik Marklund
-	Learning Long-term Dependencies Using Cognitive Inductive Biases in Self-attention RNNs (Poster) » Video http://slideslive.com/38931335 Link »	Giancarlo Kerg
-	A Differentiable Newton Euler Algorithm for Multi-body Model Learning (Poster) » Video http://slideslive.com/38931342 Link »	Michael Lutter
-	Nesterov Momentum Adversarial Perturbations in the Deep Reinforcement Learning Domain (Poster) » Video http://slideslive.com/38931437 Link »	Ezgi Korkmaz
-	DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (Poster) » Video http://slideslive.com/38931343 Link »	Aviral Kumar
-	One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL (Poster) » Video http://slideslive.com/38931345 Link »	Saurabh Kumar, Aviral Kumar
-	Attention Option-Critic (Poster) » Video http://slideslive.com/38931346 Link »	Ravi Chunduru
-	Towards Self-Paced Context Evaluation for Contextual Reinforcement Learning (Poster) » Video http://slideslive.com/38931344 Link »	Theresa Eimer
-	Group Equivariant Deep Reinforcement Learning (Poster) » Video http://slideslive.com/38931347 Link »	Arnab Kumar Mondal
-	Probing Dynamic Environments with Informed Policy Regularization (Poster) » Video http://slideslive.com/38931348 Link »	Pierre-Alexan Kamienny
-	Counterfactual Transfer via Inductive Bias in Clinical Settings (Poster) » Video http://slideslive.com/38931438 Link »	Taylor Killian
-	Learning Invariant Representations for Reinforcement Learning without Reconstruction (Poster) » Video http://slideslive.com/38931349 Link »	Amy Zhang
-	Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling (Poster) » Video http://slideslive.com/38931350 Link »	Russell Mendonca
-	Towards TempoRL: Learning When to Act (Poster) » Video http://slideslive.com/38931351 Link »	André Biedenkapp
-	On the Equivalence of Bi-Level Optimization and Game-Theoretic Formulations of Invariant Risk Minimization (Poster) » Video http://slideslive.com/38931353 Link »	Kartik Ahuja
-	PAC Imitation and Model-based Batch Learning of Contextual MDPs (Poster) » Video http://slideslive.com/38931354 Link »	Yash Nair
-	Learning Robust Representations with Score Invariant Learning (Poster) » Video http://slideslive.com/38931352 Link »	Daksh Idnani
-	Learning Off-Policy with Online Planning (Poster) » Video http://slideslive.com/38931439 Link »	Harshit Sikchi
-	Model-based Adversarial Meta-Reinforcement Learning (Poster) » Video http://slideslive.com/38931355 Link »	Tengyu Ma, Zichuan Lin
-	If MaxEnt RL is the Answer, What is the Question? (Poster) » Video http://slideslive.com/38931440 Link »	Benjamin Eysenbach
-	Multi-Task Reinforcement Learning as a Hidden-Parameter Block MDP (Poster) » Video http://slideslive.com/38931356 Link »	Amy Zhang
-	Discrete Planning with End-to-end Trained Neuro-algorithmic Policies (Poster) » Video http://slideslive.com/38931357 Link »	Marin Vlastelica Pogancic
-	Maximum Entropy Model Rollouts: Fast Model Based Policy Optimization without Compounding Errors (Poster) » Video http://slideslive.com/38931358 Link »	Chi Zhang
-	Counterfactual Data Augmentation using Locally Factored Dynamics (Poster) » Video http://slideslive.com/38931359 Link »	Silviu Pitis
-	Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (Poster) » Video http://slideslive.com/38931361 Link »	Ilya Kostrikov

Keywords: reinforcement learning · inductive bias · generalization

Author Information

Anirudh Goyal (Université de Montréal)

Rosemary Nan Ke (MILA, University of Montreal)

I am a PhD student at Mila, I am advised by Chris Pal and Yoshua Bengio. My research interest are efficient credit assignment, causal learning and model-based reinforcement learning. Here is my homepage https://nke001.github.io/

Stefan Bauer (Max Planck Institute for Intelligent Systems)

Jane Wang (DeepMind)

Theo Weber (DeepMind)

Fabio Viola (DeepMind)

Bernhard Schölkopf (MPI for Intelligent Systems Tübingen, Germany)

Bernhard Scholkopf received degrees in mathematics (London) and physics (Tubingen), and a doctorate in computer science from the Technical University Berlin. He has researched at AT&T Bell Labs, at GMD FIRST, Berlin, at the Australian National University, Canberra, and at Microsoft Research Cambridge (UK). In 2001, he was appointed scientific member of the Max Planck Society and director at the MPI for Biological Cybernetics; in 2010 he founded the Max Planck Institute for Intelligent Systems. For further information, see www.kyb.tuebingen.mpg.de/~bs.