Timezone: »

 
Workshop
1st Workshop on Language in Reinforcement Learning (LaReL)
Nantas Nardelli · Jelena Luketina · Nantas Nardelli · Jakob Foerster · Victor Zhong · Jacob Andreas · Tim Rocktäschel · Edward Grefenstette · Tim Rocktäschel

Sat Jul 18 07:00 AM -- 02:10 PM (PDT) @ None
Event URL: https://larel-ws.github.io/ »

Language is one of the most impressive human accomplishments and is believed to be the core to our ability to learn, teach, reason and interact with others. Yet, current state-of-the-art reinforcement learning agents are unable to use or understand human language at all. The ability to integrate and learn from language, in addition to rewards and demonstrations, has the potential to improve the generalization, scope and sample efficiency of agents. Furthermore, many real-world tasks, including personal assistants and general household robots, require agents to process language by design, whether to enable interaction with humans, or simply use existing interfaces. The aim of our workshop is to advance this emerging field of research by bringing together researchers from several diverse communities to discuss recent developments in relevant research areas such as instruction following and embodied language learning, and identify the most important challenges and promising research avenues.

Sat 7:00 a.m. - 7:10 a.m. [iCal]
Welcome Remarks
Sat 7:10 a.m. - 7:40 a.m. [iCal]

The ability to cooperate through language is a defining feature of humans. As the perceptual, motory and planning capabilities of deep artificial networks increase, researchers are studying whether they also can develop a shared language to interact. In this talk, I will highlight recent advances in this field but also common headaches (or perhaps limitations) with respect to experimental setup and evaluation of emergent communication. Towards making multi-agent communication a building block of human-centric AI, and by drawing from my own recent work, I will discuss approaches on making emergent communication relevant for human-agent communication in natural language.

Angeliki Lazaridou
Sat 7:40 a.m. - 8:10 a.m. [iCal]

I will discuss our progress on a research program aimed at building a Minecraft assistant. I will cover the tools and platform we have built allowing players to interact with the agents and to record those interactions, and the data we have collected. I will also cover the design of our current agent, from which we (and hopefully others) can iterate.

Nantas Nardelli
Sat 8:10 a.m. - 8:30 a.m. [iCal]
Coffee break 1 (Break)
Sat 8:30 a.m. - 9:15 a.m. [iCal]

Check out the papers and their short presentations here: https://larel-ws.github.io/accepted-papers/

Meet the authors in LaReL's Gather Town: https://tinyurl.com/gather-larel

Nantas Nardelli
Sat 9:15 a.m. - 10:15 a.m. [iCal]
Lunch Break (Break)
Sat 10:15 a.m. - 10:45 a.m. [iCal]

Models like BERT or GPT-2 can do amazing things with language, and this raises the interesting question of whether such text-based models could ever really "understand" it. One clear difference between BERT-understanding and human understanding is that BERT doesn't learn to connect language to its actions or its perception of the world it inhabits. I'll discuss an alternative approach to language understanding in which a neural-network-based agent is trained to associate words and phrases with things that it learns to see and do. First, I'll provide some evidence for the promise of this approach by showing that the interactive, first-person perspective of an agent affords it with a particular inductive bias that helps it to extend its training experience to generalize to out-of-distribution settings in ways that seem natural or 'systematic'. Second, I'll show the amount of 'propositional' (i.e. linguistic) knowledge that emerges in the internal states of the agent as it interacts with the world can be increased significantly by it learning to make predictions about observations multiple timesteps into the future. This underlines some important common ground between the agent-based and BERT-style approaches: both attest to the power of prediction and the importance of context in acquiring semantic representations. Finally, I'll connect BERT and agent-based learning in a more literal way, by showing how an agent endowed with BERT representations can achieve substantial (zero-shot) transfer from template-based language to noisy natural instructions given by humans with access to the agent's world.

Feilx Hill
Sat 10:45 a.m. - 11:15 a.m. [iCal]

In recent years, reinforcement learning (RL) has been used with considerable success in games and robotics as well as language understanding applications like dialog systems. However, the question of what language can provide for RL remains relatively under-explored. In this talk, I make the case that leveraging language will be essential to developing general-purpose interactive agents that can perform more than a single task and operate in scenarios beyond the ones they are trained on. Natural language allows us to incorporate more semantic structure into the RL framework while also making it easier to obtain guidance from humans. Specifically, I will show how several parts of the traditional RL setup (e.g. transitions, rewards, actions, goals) can be expressed in language to build agents that can handle combinatorially large spaces as well as generalize to unseen subspaces in each of these aspects.

Karthik Narasimhan
Sat 11:15 a.m. - 11:45 a.m. [iCal]

I will discuss the task of executing natural language instructions with a physical robotic agent. In contrast to existing work, we do not engineer formal representations of language meaning or the robot environment. Instead, we learn to directly map raw observations and language to low-level continuous control of a quadcopter drone. We use an interpretable neural network model that mixes learned representations with differentiable geometric operations. For training, we introduce Supervised and Reinforcement Asynchronous Learning (SuReAL), a learning algorithm that utilizes supervised and reinforcement learning processes that constantly interact to learn robust reasoning with limited data. Our learning algorithm uses demonstrations and a plan-following intrinsic reward signal. While we do not require any real-world autonomous flight during learning, our model works effectively both in simulation and the real environment.

Yoav Artzi
Sat 11:45 a.m. - 12:05 p.m. [iCal]
Coffee break 2 (Break)
Sat 12:05 p.m. - 12:50 p.m. [iCal]

Check out the papers and their short presentations here: https://larel-ws.github.io/accepted-papers/

Meet the authors in LaReL's Gather Town: https://tinyurl.com/gather-larel

Nantas Nardelli
Sat 12:50 p.m. - 1:00 p.m. [iCal]
Short Break (Break)
Sat 1:00 p.m. - 1:30 p.m. [iCal]

Text-based games are complex, interactive simulations in which text describes the game state and players make progress by entering text commands. They are fertile ground for language-focused machine learning research. In addition to language understanding, successful play requires skills like long-term memory and planning, exploration (trial and error), and common sense. The talk will introduce TextWorld, a sandbox learning environment for the training and evaluation of RL agents on text-based games. Its generative mechanisms give precise control over the difficulty, scope, and language of constructed games, and can be used to study generalization and transfer learning. This talk will also give an overview of the recent attempts to solve text-based games either using reinforcement learning or more handcrafted approaches.

Marc-Alexandre Côté
Sat 1:30 p.m. - 2:00 p.m. [iCal]

Understanding, learning and reasoning with abstract relations, like same and different or bigger and smaller, is challenging. We show that in an RL like causal learning task, very young children, 18-30 month olds, can learn both same and different relations and the functions becoming bigger and becoming smaller, generalize those relations to brand new and perceptually different objects, and use them to solve novel tasks. We suggest that both abstract causal representations, similar to causal graphical models, and early language may support this knowledge and learning.

Nantas Nardelli
Sat 2:00 p.m. - 2:10 p.m. [iCal]
Closing Remarks

Author Information

Nantas Nardelli (University of Oxford)
Jelena Luketina (The University of Oxford)
Nantas Nardelli (University of Oxford)
Jakob Foerster (Facebook)
Victor Zhong (University of Washington)
Jacob Andreas (MIT)
Tim Rocktäschel (Facebook, UCL)
Edward Grefenstette (Facebook, UCL)
Tim Rocktäschel (Facebook AI Research & University College London)

More from the Same Authors