Timezone: »

 
Oral
Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning
Jakob Foerster · Francis Song · Edward Hughes · Neil Burch · Iain Dunning · Shimon Whiteson · Matthew Botvinick · Michael Bowling

Thu Jun 13 04:40 PM -- 05:00 PM (PDT) @ Hall B
in Bandits and Multiagent Learning »

When observing the actions of others, humans make inferences about why the others acted as they did, and what this implies about their view of the world. Humans also use the fact that their actions will be interpreted in this manner when observed by others, allowing them to act informatively and thereby communicate efficiently with others. Although learning algorithms have recently achieved superhuman performance in a number of two-player, zero-sum games, scalable multi-agent reinforcement learning algorithms that can discover effective strategies and conventions in complex, partially observable settings have proven elusive. We present the \emph{Bayesian action decoder} (BAD), a new multi-agent learning method that uses an approximate Bayesian update to obtain a public belief that conditions on the actions taken by all agents in the environment. Together with the public belief, this Bayesian update effectively defines a new Markov decision process, the \emph{public belief MDP}, in which the action space consists of deterministic partial policies, parameterised by neural networks, that can be sampled for a given public state. BAD exploits the fact that an agent acting only on this public belief state can still learn to use its private information if the action space is augmented to be over partial policies mapping private information into environment actions. The Bayesian update is also closely related to the \emph{theory of mind} reasoning that humans carry out when observing others' actions. We first validate BAD on a proof-of-principle two-step matrix game, where it outperforms policy gradient methods. We then evaluate BAD on the challenging, cooperative partial-information card game Hanabi, where in the two-player setting the method surpasses all previously published learning and hand-coded approaches.

Author Information

Jakob Foerster (Facebook AI Research)
Jakob Foerster

Jakob Foerster started as an Associate Professor at the department of engineering science at the University of Oxford in the fall of 2021. During his PhD at Oxford he helped bring deep multi-agent reinforcement learning to the forefront of AI research and interned at Google Brain, OpenAI, and DeepMind. After his PhD he worked as a research scientist at Facebook AI Research in California, where he continued doing foundational work. He was the lead organizer of the first Emergent Communication workshop at NeurIPS in 2017, which he has helped organize ever since and was awarded a prestigious CIFAR AI chair in 2019. His past work addresses how AI agents can learn to cooperate and communicate with other agents, most recently he has been developing and addressing the zero-shot coordination problem setting, a crucial step towards human-AI coordination.

Francis Song (DeepMind)
Edward Hughes (DeepMind)
Neil Burch (DeepMind)
Iain Dunning (DeepMind)
Shimon Whiteson (University of Oxford)
Matthew Botvinick (DeepMind)
Michael Bowling (DeepMind)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors