Timezone: »
One of the most significant and challenging open problems in Artificial Intelligence (AI) is the problem of Lifelong Learning. Lifelong Machine Learning considers systems that can continually learn many tasks (from one or more domains) over a lifetime. A lifelong learning system efficiently and effectively:
1. retains the knowledge it has learned from different tasks;
2. selectively transfers knowledge (from previously learned tasks) to facilitate the learning of new tasks;
3. ensures the effective and efficient interaction between (1) and (2).
Lifelong Learning introduces several fundamental challenges in training models that generally do not arise in a single task batch learning setting. This includes problems like catastrophic forgetting and capacity saturation. This workshop aims to explore solutions for these problems in both supervised learning and reinforcement learning settings.
|
Sat 2:00 a.m. - 2:15 a.m.
|
Opening Comments
(Introduction)
Opening Remarks (Introduction and Overview) |
Sarath Chandar · Shagun Sodhani 🔗 |
|
Sat 2:15 a.m. - 2:45 a.m.
|
Challenges & Opportunities in Lifelong Reinforcement Learning by Katja Hoffman
(Talk)
SlidesLive Video » Lifelong reinforcement learning holds promise to enable autonomous decision making in applications ranging from household robotics to autonomous space exploration. In this talk we will discuss key challenges that need to be address to make progress towards this grand vision. First, we discuss the fundamental need for the community to invest in a shared understanding of problem formalizations and evaluation paradigms and benchmarks. Second, we dive deeper into ongoing work towards tackling exploration and representation in lifelong reinforcement learning. Our aim is to spark debate and inspire research in this exciting space. |
Katja Hofmann · Rika Antonova · Luisa Zintgraf 🔗 |
|
Sat 2:45 a.m. - 3:00 a.m.
|
Q&A with Katja Hoffman
(Q&A)
|
Katja Hofmann · Luisa Zintgraf · Rika Antonova · Sarath Chandar · Shagun Sodhani 🔗 |
|
Sat 3:00 a.m. - 3:30 a.m.
|
Never-ending Learning by Partha Pratim Talukdar
(Talk)
Humans learn many different functions and skills, from diverse experiences gained over many years, from a staged curriculum in which they first learn easier and later more difficult tasks, retain the learned knowledge and skills, which are used in subsequent learning to make it easier or more effective. Furthermore, humans self-reflect on their evolving skills, choose new learning tasks over time, teach one another, learn new representations, read books, discuss competing hypotheses, and more. In this talk, I shall share my thoughts on the question of how to design machine learning agents with similar capabilities. |
Partha Talukdar 🔗 |
|
Sat 3:30 a.m. - 3:45 a.m.
|
Q&A with Partha Pratim Talukdar
(Q&A)
|
Partha Talukdar · Shagun Sodhani · Sarath Chandar 🔗 |
|
Sat 3:45 a.m. - 4:00 a.m.
|
Contributed Talk: Continual Deep Learning by Functional Regularisation of Memorable Past
(Talk)
SlidesLive Video » |
Siddharth Swaroop 🔗 |
|
Sat 4:00 a.m. - 6:00 a.m.
|
Virtual Poster Session #1
(Poster Session)
|
🔗 |
|
Sat 6:00 a.m. - 6:30 a.m.
|
Credit assignment and meta-learning in a single lifelong trial by Jürgen Schmidhuber
(Talk)
SlidesLive Video » Most current artificial reinforcement learning (RL) agents are trained under the assumption of repeatable trials, and are reset at the beginning of each trial. Humans, however, are never reset. Instead, they are allowed to discover computable patterns across trials, e.g.: in every third trial, go left to obtain reward, otherwise go right. General RL (sometimes called AGI) must assume a single lifelong trial which may or may not include identifiable sub-trials. General RL must also explicitly take into account that policy changes in early life may affect properties of later sub-trials and policy changes. In particular, General RL must take into account recursively that early meta-meta-learning is setting the stage for later meta-learning which is setting the stage for later learning etc. Most popular RL mechanisms, however, ignore such lifelong credit assignment chains. Exceptions are the success story algorithm (1990s), AIXI (2000s), and the mathematically optimal Gödel Machine (2003). |
Jürgen Schmidhuber 🔗 |
|
Sat 6:30 a.m. - 6:45 a.m.
|
Q&A with Jürgen Schmidhuber
(Q&A)
|
Jürgen Schmidhuber · Shagun Sodhani · Sarath Chandar 🔗 |
|
Sat 6:45 a.m. - 7:00 a.m.
|
Contributed Talk: Combining Variational Continual Learning with FiLM Layers
(Talk)
SlidesLive Video » |
Noel Loo 🔗 |
|
Sat 7:00 a.m. - 7:15 a.m.
|
Contributed Talk: Wandering Within a World: Online Contextualized Few-Shot Learning
(Talk)
|
Mengye Ren 🔗 |
|
Sat 7:15 a.m. - 7:45 a.m.
|
Invited Talk: Lifelong Learning: Towards Broad and Robust AI by Irina Rish
(Talk)
Modern AI systems have achieved impressive results in many specific domains, from image and speech recognition to natural language processing and mastering complex games such as chess and Go. However, they remain largely inflexible, fragile and narrow, unable to continually adapt to a wide range of changing environments and novel tasks without "catastrophically forgetting" what they have learned before, to infer higher-order abstractions allowing for systematic generalization to out-of-distribution data, and to achieve the level of robustness necessary to "survive" various perturbations in their environment - a natural property of most biological intelligent systems. In this talk, we will provide a brief overview of advances in continual learning (CL) field [1] which aims to push AI from "narrow" to "broad", from unsupervised adaptive ("neurogenetic") architectural adaptations [2] to a recent general supervised CL framework for quickly solving new, out-of-distribution tasks, combined with fast remembering of the previous ones; it unifies continual-, meta-, meta-continual-, and continual-meta learning and introduces continual-MAML, an online extension of the popular MAML algorithm [3]. Furthermore, we present a brief overview of the most challenging setting - continual RL, characterized by dynamic, non-stationary environment, and discuss open problems and challenges in bridging the gap between the current state of continual RL and better incremental reinforcement learners that can function in increasingly human realistic learning environments [4]. Next, we address the robust representation learning problem, i.e. extracting features invariant to various stochastic and/or adversarial perturbations of the environment - a common goal across continual-, meta-, transfer learning as well as adversarial robustness, out-of-distribution generalization, self-supervised learning, and related subfields. As an example, our recent Adversarial Feature Desensitization (AFD) approach [5] trains a feature extractor network to generate representations which are both predictive and robust to input perturbations (e.g. adversarial attacks) and demonstrates a significant improvement over the state-of-the-art, despite its relative simplicity (i.e., feature robustness is enforced via additional adversarial decoder with a GAN-like objective attempting to discriminate between the original and perturbed inputs). Finally, we conclude the talk with a discussion of severa directions for future work, which including drawing inspirations (e.g., inductive biases) from neuroscience [6], in order to develop truly broad and robust lifelong-learning AI systems. Related work:
[1] https://arxiv.org/abs/1909.08383 de Lange et al (2019) A continual learning survey: Defying forgetting in classification tasks.
[2] https://arxiv.org/abs/1701.06106 Garg et al (2017). Neurogenesis-Inspired Dictionary Learning: Online Model Adaptation in a Changing World. IJCAI 2017.
[3] https://arxiv.org/abs/2003.05856 Caccia et al (2020). Online Fast Adaptation and Knowledge Accumulation: a New Approach to Continual Learning. submitted.
[4] (in preparation) Khetarpal et al (2020). Towards Continual Reinforcement Learning: A Review and Perspectives. |
Irina Rish 🔗 |
|
Sat 7:45 a.m. - 8:00 a.m.
|
Q&A with Irina Rish
(Q&A)
|
Irina Rish · Shagun Sodhani · Sarath Chandar 🔗 |
|
Sat 8:00 a.m. - 9:00 a.m.
|
Virtual Poster Session #2
(Poster Session)
|
🔗 |
|
Sat 9:00 a.m. - 9:15 a.m.
|
Contributed Talk: Deep Reinforcement Learning amidst Lifelong Non-Stationarity
(Talk)
|
Annie Xie 🔗 |
|
Sat 9:15 a.m. - 9:30 a.m.
|
Contributed Talk: Lifelong Learning of Factored Policies via Policy Gradients
(Talk)
|
Jorge Mendez Mendez 🔗 |
|
Sat 9:30 a.m. - 10:00 a.m.
|
The FOAK Cycle for Model-based Life-long Learning by Rich Sutton
(Talk)
A life-long learning agent should learn not only to solve problems, but also to pose new problems for itself. In reinforcement learning, the starting problems are maximizing reward and predicting value, and the natural new problems are achieving subgoals and predicting what will happen next. There has been a lot of work that provides a language for learning new problems (e.g., on auxiliary tasks and general value functions), but precious little that actually learns them (e.g., McGovern on learning subgoal states). In this talk I present a general strategy for learning new problems and, moreover, for learning an endless cycle of problems and solutions, each leading to the other. I call this cycle the FOAK cycle, because it is based on Features, Options, And Knowledge, where “options” are temporally extended ways of behaving, and “knowledge” refers to an agent’s option-conditional model of the transition dynamics of the world. The new problems in the FOAK cycle are 1) to find options that attain state features and 2) to model the consequences of those options. As these problems are solved and the models are used in planning, more abstract features are formed and made the basis for new options and models, continuing the cycle. The FOAK cycle is intended to produce a model-based reinforcement learning agent with successively more abstract representations and knowledge of its world, in other words, a life-long learning agent. |
Richard Sutton 🔗 |
|
Sat 10:00 a.m. - 10:15 a.m.
|
Q&A by Rich Sutton
(Q&A)
|
Richard Sutton · Shagun Sodhani · Sarath Chandar 🔗 |
|
Sat 10:15 a.m. - 10:30 a.m.
|
Contributed Talk: Gradient Based Memory Editing for Task-Free Continual Learning
(Talk)
|
Xisen Jin 🔗 |
|
Sat 10:30 a.m. - 11:30 a.m.
|
Panel Discussion
(Discussion Panel)
You can submit questions for the panel at https://app.sli.do/event/xf5illav/live/questions |
Eric Eaton · Martha White · Doina Precup · Irina Rish · Harm van Seijen 🔗 |
|
Sat 11:30 a.m. - 11:45 a.m.
|
Concluding Remarks
|
Sarath Chandar · Shagun Sodhani 🔗 |
|
-
|
Accepted Papers
(Demonstration)
|
🔗 |
Author Information
Shagun Sodhani (Facebook AI Research)
Sarath Chandar (Polytechnique Montreal)
Balaraman Ravindran (Indian Institute of Technology)
Doina Precup (McGill University / DeepMind)
More from the Same Authors
-
2021 : Randomized Least Squares Policy Optimization »
Haque Ishfaq · Zhuoran Yang · Andrei Lupu · Viet Nguyen · Lewis Liu · Riashat Islam · Zhaoran Wang · Doina Precup -
2021 : Finite time analysis of temporal difference learning with linear function approximation: the tail averaged case »
Gandharv Patil · Prashanth L.A. · Doina Precup -
2022 Workshop: Decision Awareness in Reinforcement Learning »
Evgenii Nikishin · Pierluca D'Oro · Doina Precup · Andre Barreto · Amir-massoud Farahmand · Pierre-Luc Bacon -
2022 Poster: Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error »
Scott Fujimoto · David Meger · Doina Precup · Ofir Nachum · Shixiang Gu -
2022 Poster: Robust Policy Learning over Multiple Uncertainty Sets »
Annie Xie · Shagun Sodhani · Chelsea Finn · Joelle Pineau · Amy Zhang -
2022 Spotlight: Robust Policy Learning over Multiple Uncertainty Sets »
Annie Xie · Shagun Sodhani · Chelsea Finn · Joelle Pineau · Amy Zhang -
2022 Spotlight: Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error »
Scott Fujimoto · David Meger · Doina Precup · Ofir Nachum · Shixiang Gu -
2022 Poster: The Neural Race Reduction: Dynamics of Abstraction in Gated Networks »
Andrew Saxe · Shagun Sodhani · Sam Lewallen -
2022 Poster: Improving Robustness against Real-World and Worst-Case Distribution Shifts through Decision Region Quantification »
Leo Schwinn · Leon Bungert · An Nguyen · René Raab · Falk Pulsmeyer · Doina Precup · Bjoern Eskofier · Dario Zanca -
2022 Poster: Towards Evaluating Adaptivity of Model-Based Reinforcement Learning Methods »
Yi Wan · Ali Rahimi-Kalahroudi · Janarthanan Rajendran · Ida Momennejad · Sarath Chandar · Harm van Seijen -
2022 Spotlight: Towards Evaluating Adaptivity of Model-Based Reinforcement Learning Methods »
Yi Wan · Ali Rahimi-Kalahroudi · Janarthanan Rajendran · Ida Momennejad · Sarath Chandar · Harm van Seijen -
2022 Spotlight: The Neural Race Reduction: Dynamics of Abstraction in Gated Networks »
Andrew Saxe · Shagun Sodhani · Sam Lewallen -
2022 Spotlight: Improving Robustness against Real-World and Worst-Case Distribution Shifts through Decision Region Quantification »
Leo Schwinn · Leon Bungert · An Nguyen · René Raab · Falk Pulsmeyer · Doina Precup · Bjoern Eskofier · Dario Zanca -
2021 Poster: Randomized Exploration in Reinforcement Learning with General Value Function Approximation »
Haque Ishfaq · Qiwen Cui · Viet Nguyen · Alex Ayoub · Zhuoran Yang · Zhaoran Wang · Doina Precup · Lin Yang -
2021 Spotlight: Randomized Exploration in Reinforcement Learning with General Value Function Approximation »
Haque Ishfaq · Qiwen Cui · Viet Nguyen · Alex Ayoub · Zhuoran Yang · Zhaoran Wang · Doina Precup · Lin Yang -
2021 Poster: Locally Persistent Exploration in Continuous Control Tasks with Sparse Rewards »
Susan Amin · Maziar Gomrokchi · Hossein Aboutalebi · Harsh Satija · Doina Precup -
2021 Poster: A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representation »
Scott Fujimoto · David Meger · Doina Precup -
2021 Poster: Multi-Task Reinforcement Learning with Context-based Representations »
Shagun Sodhani · Amy Zhang · Joelle Pineau -
2021 Spotlight: A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representation »
Scott Fujimoto · David Meger · Doina Precup -
2021 Spotlight: Locally Persistent Exploration in Continuous Control Tasks with Sparse Rewards »
Susan Amin · Maziar Gomrokchi · Hossein Aboutalebi · Harsh Satija · Doina Precup -
2021 Spotlight: Multi-Task Reinforcement Learning with Context-based Representations »
Shagun Sodhani · Amy Zhang · Joelle Pineau -
2021 Poster: Preferential Temporal Difference Learning »
Nishanth Anand · Doina Precup -
2021 Spotlight: Preferential Temporal Difference Learning »
Nishanth Anand · Doina Precup -
2020 : Concluding Remarks »
Sarath Chandar · Shagun Sodhani -
2020 : Panel Discussion »
Eric Eaton · Martha White · Doina Precup · Irina Rish · Harm van Seijen -
2020 : Q&A by Rich Sutton »
Richard Sutton · Shagun Sodhani · Sarath Chandar -
2020 : Q&A with Irina Rish »
Irina Rish · Shagun Sodhani · Sarath Chandar -
2020 : Q&A with Jürgen Schmidhuber »
Jürgen Schmidhuber · Shagun Sodhani · Sarath Chandar -
2020 : Q&A with Partha Pratim Talukdar »
Partha Talukdar · Shagun Sodhani · Sarath Chandar -
2020 : Q&A with Katja Hoffman »
Katja Hofmann · Luisa Zintgraf · Rika Antonova · Sarath Chandar · Shagun Sodhani -
2020 : Opening Comments »
Sarath Chandar · Shagun Sodhani -
2020 Poster: Interference and Generalization in Temporal Difference Learning »
Emmanuel Bengio · Joelle Pineau · Doina Precup -
2020 Poster: Invariant Causal Prediction for Block MDPs »
Amy Zhang · Clare Lyle · Shagun Sodhani · Angelos Filos · Marta Kwiatkowska · Joelle Pineau · Yarin Gal · Doina Precup -
2020 : Mentoring Panel: Doina Precup, Deborah Raji, Anima Anandkumar, Angjoo Kanazawa and Sinead Williamson (moderator). »
Doina Precup · Inioluwa Raji · Angjoo Kanazawa · Sinead A Williamson · Animashree Anandkumar -
2020 : Invited Talk: Doina Precup on Building Knowledge for AI Agents with Reinforcement Learning »
Doina Precup -
2019 Workshop: Workshop on Multi-Task and Lifelong Reinforcement Learning »
Sarath Chandar · Shagun Sodhani · Khimya Khetarpal · Tom Zahavy · Daniel J. Mankowitz · Shie Mannor · Balaraman Ravindran · Doina Precup · Chelsea Finn · Abhishek Gupta · Amy Zhang · Kyunghyun Cho · Andrei A Rusu · Facebook Rob Fergus -
2019 : Networking Lunch (provided) + Poster Session »
Abraham Stanway · Alex Robson · Aneesh Rangnekar · Ashesh Chattopadhyay · Ashley Pilipiszyn · Benjamin LeRoy · Bolong Cheng · Ce Zhang · Chaopeng Shen · Christian Schroeder · Christian Clough · Clement DUHART · Clement Fung · Cozmin Ududec · Dali Wang · David Dao · di wu · Dimitrios Giannakis · Dino Sejdinovic · Doina Precup · Duncan Watson-Parris · Gege Wen · George Chen · Gopal Erinjippurath · Haifeng Li · Han Zou · Herke van Hoof · Hillary A Scannell · Hiroshi Mamitsuka · Hongbao Zhang · Jaegul Choo · James Wang · James Requeima · Jessica Hwang · Jinfan Xu · Johan Mathe · Jonathan Binas · Joonseok Lee · Kalai Ramea · Kate Duffy · Kevin McCloskey · Kris Sankaran · Lester Mackey · Letif Mones · Loubna Benabbou · Lynn Kaack · Matthew Hoffman · Mayur Mudigonda · Mehrdad Mahdavi · Michael McCourt · Mingchao Jiang · Mohammad Mahdi Kamani · Neel Guha · Niccolo Dalmasso · Nick Pawlowski · Nikola Milojevic-Dupont · Paulo Orenstein · Pedram Hassanzadeh · Pekka Marttinen · Ramesh Nair · Sadegh Farhang · Samuel Kaski · Sandeep Manjanna · Sasha Luccioni · Shuby Deshpande · Soo Kim · Soukayna Mouatadid · Sunghyun Park · Tao Lin · Telmo Felgueira · Thomas Hornigold · Tianle Yuan · Tom Beucler · Tracy Cui · Volodymyr Kuleshov · Wei Yu · yang song · Ydo Wexler · Yoshua Bengio · Zhecheng Wang · Zhuangfang Yi · Zouheir Malki -
2019 Poster: Off-Policy Deep Reinforcement Learning without Exploration »
Scott Fujimoto · David Meger · Doina Precup -
2019 Oral: Off-Policy Deep Reinforcement Learning without Exploration »
Scott Fujimoto · David Meger · Doina Precup -
2018 Poster: Convergent Tree Backup and Retrace with Function Approximation »
Ahmed Touati · Pierre-Luc Bacon · Doina Precup · Pascal Vincent -
2018 Oral: Convergent Tree Backup and Retrace with Function Approximation »
Ahmed Touati · Pierre-Luc Bacon · Doina Precup · Pascal Vincent -
2017 : Panel Discussion »
Balaraman Ravindran · Chelsea Finn · Alessandro Lazaric · Katja Hofmann · Marc Bellemare -
2017 : Some experiments with learning hyperparameters, transfer, and multi-task leaning »
Balaraman Ravindran -
2017 Workshop: Reinforcement Learning Workshop »
Doina Precup · Balaraman Ravindran · Pierre-Luc Bacon -
2017 : Lifelong Learning - Panel Discussion »
Sergey Levine · Joelle Pineau · Balaraman Ravindran · Andrei A Rusu -
2017 Workshop: Lifelong Learning: A Reinforcement Learning Approach »
Sarath Chandar · Balaraman Ravindran · Daniel J. Mankowitz · Shie Mannor · Tom Zahavy