Timezone: »

 
Oral
Human-Timescale Adaptation in an Open-Ended Task Space
Jakob Bauer · Kate Baumli · Feryal Behbahani · Avishkar Bhoopchand · Natalie Bradley-Schmieg · Michael Chang · Natalie Clay · Adrian Collister · Vibhavari Dasagi · Lucy Gonzalez · Karol Gregor · Edward Hughes · Sheleem Kashem · Maria Loks-Thompson · Hannah Openshaw · Jack Parker-Holder · Shreya Pathak · Nicolas Perez-Nieves · Nemanja Rakicevic · Tim Rocktäschel · Yannick Schroecker · Satinder Singh · Jakub Sygnowski · Karl Tuyls · Sarah York · Alexander Zacherl · Lei Zhang

Tue Jul 25 09:34 PM -- 09:42 PM (PDT) @ Ballroom C
in Oral A6 Reinforcement Learning 2 »

Foundation models have shown impressive adaptation and scalability in supervised and self-supervised learning problems, but so far these successes have not fully translated to reinforcement learning (RL). In this work, we demonstrate that training an RL agent at scale leads to a general in-context learning algorithm that can adapt to open-ended novel embodied 3D problems as quickly as humans. In a vast space of held-out environment dynamics, our adaptive agent (AdA) displays on-the-fly hypothesis-driven exploration, efficient exploitation of acquired knowledge, and can successfully be prompted with first-person demonstrations. Adaptation emerges from three ingredients: (1) meta-reinforcement learning across a vast, smooth and diverse task distribution, (2) a policy parameterised as a large-scale attention-based memory architecture, and (3) an effective automated curriculum that prioritises tasks at the frontier of an agent's capabilities. We demonstrate characteristic scaling laws with respect to network size, memory length, and richness of the training task distribution. We believe our results lay the foundation for increasingly general and adaptive RL agents that perform well across ever-larger open-ended domains.

[ PDF] 

Author Information

Jakob Bauer (DeepMind)
Kate Baumli (Google DeepMind)
Feryal Behbahani (Google DeepMind)
Avishkar Bhoopchand (DeepMind)
Natalie Bradley-Schmieg
Michael Chang
Natalie Clay (DeepMind)
Adrian Collister
Vibhavari Dasagi (DeepMind)
Lucy Gonzalez
Karol Gregor (Google)
Edward Hughes (DeepMind)
Sheleem Kashem
Maria Loks-Thompson
Hannah Openshaw
Jack Parker-Holder (DeepMind)
Shreya Pathak
Nicolas Perez-Nieves (Imperial College London)
Nemanja Rakicevic (DeepMind)
Tim Rocktäschel (Facebook AI Research & University College London)
Yannick Schroecker (DeepMind)
Satinder Singh (DeepMind)
Jakub Sygnowski (Google)
Karl Tuyls (DeepMind)
Sarah York
Alexander Zacherl (DeepMind)
Lei Zhang (Google DeepMind)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors