ICML 2020

Timezone: »

Workshop

Workshop on Learning in Artificial Open Worlds

Arthur Szlam · Katja Hofmann · Ruslan Salakhutdinov · Noboru Kuno · William Guss · Kavya Srinet · Brandon Houghton

Sat Jul 18 07:00 AM -- 02:00 PM (PDT) @

in Workshops Sat »

Event URL: https://sites.google.com/view/icml-laow2020/home »

In situations where a task can be cleanly formulated and data is plentiful, modern machine learning (ML) techniques have achieved impressive (and often super-human) results. Here, plentiful data'' can mean labels from humans, access to a simulator and well designed reward function, or other forms of interaction and supervision. On the other hand, in situations where tasks cannot be cleanly formulated and plentifully supervised, ML has not yet shown the same progress. We still seem far from flexible agents that can learn without human engineers carefully designing or collating their supervision. This is problematic in many settings where machine learning is or will be applied in real world settings, where these agents have to interact with human users and may be used in settings that go beyond any initial clean training data used during system development. A key open question is how to make machine learning effective and robust enough to operate in real world open domains. Artificial {\it open} worlds are ideal laboratories for studying how to extend the successes of ML to build such agents. Open worlds are characterized by: \begin{itemize} \item Large (or perhaps infinite) collections of tasks, often not specified till test time; or lack of well defined tasks altogether (despite there being lots to do). \itemunbounded'' environments, long ``epsiodes''; or no episodes at all.
\item Many interacting agents; more generally, emergent behavior from interactions with the environment.
\end{itemize}
On one hand, they retain many of the challenging features of the real world with respect to studying learning agents. On the other hand,
they allow cheap collection of environment interaction data. Furthermore, because many artificial worlds of interest are games that people enjoy playing, they could allow interaction with humans at scale.

A particularly promising direction is that open world games can bridge: the closed domains and benchmarks that have traditionally driven research progress and open ended real world applications in which resulting technology is deployed.
We propose a workshop designed to catalyze research towards addressing these challenges posed by machine learning in open worlds.
Our goal is to bring together researchers with a wide range of perspectives whose work focuses on, or is enabled by, open worlds. This would be the very first workshop focused on this topic, and we anticipate that it would play a key role in sharing experience, brainstorming ideas, and catalyzing novel directions for research.

Sat 7:00 a.m. - 7:15 a.m.	Opening remarks	Katja Hofmann 🔗
Sat 7:15 a.m. - 7:35 a.m.	The NetHack Learning Environment ( Invited talk 1 ) SlidesLive Video » Progress in Reinforcement Learning (RL) algorithms goes hand-in-hand with the development of challenging environments that test the limits of current methods. While existing RL environments are either sufficiently complex or based on fast simulation, they are rarely both these things. In this talk, I present the NetHack Learning Environment (NLE), a scalable, procedurally generated, stochastic, rich, and challenging environment for RL research based on the popular single-player terminal-based rogue like game, NetHack. We argue that NetHack is sufficiently complex to drive long-term research on problems such as exploration, planning, skill acquisition, and language-conditioned RL, while dramatically reducing the computational resources required to gather a large amount of experience. We compare NLE and its task suite to existing alternatives, and discuss why it is an ideal medium for testing the robustness and systematic generalization of RL agents. We demonstrate empirical success for early stages of the game using a distributed deep RL baseline and Random Network Distillation exploration, alongside qualitative analysis of various agents trained in the environment. NLE is open-source at https://github.com/facebookresearch/nle .	Tim Rocktäschel 🔗
Sat 7:35 a.m. - 7:45 a.m.	The NetHack Learning Environment Q&A ( Invited talk 1 Q&A ) Please join using the Zoom link and post your questions on Rocketchat.	Tim Rocktäschel · Katja Hofmann 🔗
Sat 7:45 a.m. - 8:05 a.m.	Open-ended environments for advancing RL ( Invited talk 2 ) The field of reinforcement learning is pushed forwards by the presence of challenging environments. Over the years, the complexity of these environments has continued to increase, but the question is how can we continue to push the complexity of environments with respect to the optimal policy complexity in a scalable manner. Here I will discuss using multi-agent environments to create more open-ended environments, and discuss examples of our work to move in this direction with Capture the Flag and Starcraft 2. Finally I will discuss some future directions for generating even more open-ended environments to further push our RL algorithms.	Max Jaderberg 🔗
Sat 8:05 a.m. - 8:15 a.m.	Open-ended environments for advancing RL Q&A ( Invited talk 2 Q&A ) Please join using the Zoom link and post your questions on Rocketchat.	Max Jaderberg · Katja Hofmann 🔗
Sat 8:15 a.m. - 8:30 a.m.	Coffee break ( coffee break )	Kavya Srinet 🔗
Sat 8:30 a.m. - 8:50 a.m.	Collaborative Construction and Communication in Minecraft ( Invited talk 3 ) I will present work done by my group on defining a collaborative construction task that allows us to use the Minecraft platform to study situated natural language generation and understanding. In this task, one player (the Architect) needs to instruct another (the Builder) via a chat interface to construct a given target structure that only the Architect is shown. I will discuss what makes this task interesting and challenging. I will also describe models that we have developed for the Architect and the Builder role, and discuss what remains to be done to create agents that can solve this task.	Julia Hockenmaier 🔗
Sat 8:50 a.m. - 9:00 a.m.	Collaborative Construction and Communication in Minecraft Q&A ( Invited talk 3 Q&A ) Please join using the Zoom link and post your questions on Rocketchat.	Julia Hockenmaier · Arthur Szlam 🔗
Sat 9:00 a.m. - 9:20 a.m.	Collaboration in Situated Instruction Following ( Invited talk 4 ) SlidesLive Video » I will focus on the problem of executing natural language instructions in a collaborative environment. I will propose the task of learning to follow sequences of instructions in a collaborative scenario, where two agents, a leader and a follower, execute actions in the environment and the leader controls the follower using natural language. To study this problem, we build CerealBar, a multi-player 3D game where a leader instructs a follower, and both act in the environment together to accomplish complex goals. I will focus on learning an autonomous follower that executes the instructions of a human leader. I will briefly describe a model to address this problem, and a learning method that relies on static recorded human-human interactions, while still learning to recover from cascading errors between instructions.	Yoav Artzi 🔗
Sat 9:20 a.m. - 9:30 a.m.	Collaboration in Situated Instruction Following Q&A ( Invited talk 4 Q&A ) Please join using the Zoom link and post your questions on Rocketchat.	Yoav Artzi · Arthur Szlam 🔗
Sat 9:30 a.m. - 10:00 a.m.	Lunch and networking	Kavya Srinet 🔗
Sat 10:00 a.m. - 10:20 a.m.	What an Agent Knows: Evaluation in Open Worlds ( Invited talk 5 ) SlidesLive Video » Agents tackling complex problems in open environments often benefit from the ability to construct knowledge. Learning to independently solve sub-tasks and form models of the world can help agents progress in solving challenging problems. In this talk, we draw attention to challenges that arise when evaluating an agent’s knowledge, specifically focusing on methods that express an agent’s knowledge as predictions. Using the General Value Function framework we highlight the distinction between useful knowledge and strict measures of accuracy. Having identified challenges in assessing an agent’s knowledge, we propose a possible evaluation approach that is compatible with large and open worlds.	Alex Kearney 🔗
Sat 10:20 a.m. - 10:30 a.m.	What an Agent Knows: Evaluation in Open Worlds Q&A ( Invited talk 5 Q&A ) Please join using the Zoom link and post your questions on Rocketchat.	Alex Kearney · William Guss 🔗
Sat 10:30 a.m. - 10:50 a.m.	Endless Frontiers? ( Invited talk 6 ) The research community is gradually coming to a realization that policies trained arcade-like video games are very limited. They overfit badly and are not going to take us far along the way to some sort of general intelligence. This should perhaps not be surprising, given that such games generally have tightly defined tasks, fixed perspectives, and generally static worlds. More and more attention is therefore given to games that are in some sense open-ended or feature open worlds. Could such games be the solution to our problems, allowing the development of more general artificial intelligence? Perhaps, but basing competitions or benchmarks on open-ended games is not going to be easy, as the very features which make for a good benchmark are the same that lead to brittle policies. Shoe-horning open-world games into a standard RL framework is unlikely to be the best option for going forward. Many of the most interesting opportunities for developing intelligent behavior is likely to come from agents constructing their own challenges and environments. The boundary between playing a game and constructing a world is not well-defined: I will give examples from where the same RL setup was used to play SimCity and to develop game levels. I will also briefly introduce the Generative Design in Minecraft Competition, which focuses on building believable settlements.	Julian Togelius 🔗
Sat 10:50 a.m. - 11:00 a.m.	Endless Frontiers? Q&A ( Invited talk 6 Q&A ) Please join using the Zoom link and post your questions on Rocketchat.	Julian Togelius · William Guss 🔗
Sat 11:00 a.m. - 11:30 a.m.	Coffee break ( coffee break )	Kavya Srinet 🔗
Sat 11:30 a.m. - 1:00 p.m.	Poster session ( Poster session and demo ) Hey all! Our poster session is hosted in a virtual town. To go to the poster session, please go to : https://gather.town/bWIYAlhNYvQPRkCE/ICML2020-LAOW and use the password: flatfish	Noboru Kuno · Kavya Srinet · William Guss · Brandon Houghton 🔗
Sat 1:00 p.m. - 2:00 p.m.	Panel discussion Join us for a panel discussion with our invited speakers	Kavya Srinet · Katja Hofmann · Yoav Artzi · Alex Kearney · Julia Hockenmaier 🔗

Keywords: open worlds · lifelong learning · domain adaptation · natural language understanding · multi-agent learning

Author Information

Arthur Szlam (Facebook)

Katja Hofmann (Microsoft)

Ruslan Salakhutdinov (Carnegie Mellen University)

Noboru Kuno (Microsoft Research)

Sean Kuno is a Senior Research Program Manager of Microsoft Research Outreach. He is based in Redmond U.S. and he is a member of Artificial Intelligence Engaged team of Microsoft Research Outreach. Kuno leads the ideation, design and launch of community programs for AI projects such as Project Malmo, working in partnership with universities and government agencies worldwide.

Author Information

Arthur Szlam (Facebook)

Katja Hofmann (Microsoft)

Ruslan Salakhutdinov (Carnegie Mellen University)

Noboru Kuno (Microsoft Research)

William Guss (Carnegie Mellon University)

Kavya Srinet (Facebook AI Research)

Brandon Houghton (OpenAI)

More from the Same Authors