Timezone: »
Many of the world's most pressing issues, such as climate change, pandemics, financial market stability and fake news, are emergent phenomena that result from the interaction between a large number of strategic or learning agents. Understanding these systems is thus a crucial frontier for scientific and technology development that has the potential to permanently improve the safety and living standards of humanity. AgentBased Modelling (ABM) (also known as individualbased modelling) is an approach toward creating simulations of these types of complex systems by explicitly modelling the actions and interactions of the individual agents contained within. However, current methodologies for calibrating and validating ABMs rely on human expert domain knowledge and handcoded behaviours for individual agents and environment dynamics. Recent progress in AI has the potential to offer exciting new approaches to learning, calibrating, validation, analysing and accelerating ABMs. This interdisciplinary workshop is meant to bring together practitioners and theorists to boost ABM method development in AI, and stimulate novel applications across disciplinary boundaries  making ICML the ideal venue.Our inaugural workshop will be organised along two axes. First, we seek to provide a venue where ABM researchers from a variety of domains can introduce AI researchers to their respective domain problems. To this end, we are inviting a number of highprofile speakers across various application domains. Second, we seek to stimulate research into AI methods that can scale to largescale agentbased models with the potential to redefine our capabilities of creating, calibrating, and validating such models. These methods include, but are not limited to, simulationbased inference, multiagent learning, causal inference and discovery, program synthesis, and the development of domainspecific languages and tools that allow for tight integration of ABMs and AI approaches.
Sat 5:30 a.m.  5:40 a.m.

Introduction by the Organizers
(Live intro)
SlidesLive Video » 
Christian Schroeder de Witt 🔗 
Sat 5:40 a.m.  6:10 a.m.

What will be the ImageNet moment for ABMs?
(Invited Talk)

Stephan Zheng 🔗 
Sat 6:10 a.m.  6:40 a.m.

The intersection of simulationbased inference and agent based modelling
(Invited Talk)
SlidesLive Video » 
Kyle Cranmer 🔗 
Sat 6:40 a.m.  7:10 a.m.

Reservoir Computing for Predicting Complex Network Dynamics
(Invited Talk)
SlidesLive Video » 
Michelle Girvan 🔗 
Sat 7:10 a.m.  7:40 a.m.

Adding AI to AgentBased Models – Applications in infectious disease epidemiology
(Invited Talk)
SlidesLive Video » 
Theresa Reiker 🔗 
Sat 7:40 a.m.  7:50 a.m.

Generating Diverse Cooperative Agents by Learning Incompatible Policies
(Spotlight (Contributed))
SlidesLive Video » Effectively training a robust agent that can cooperate with unseen agents requires diverse training partner agents. Nonetheless, obtaining cooperative agents with diverse behaviors is a challenging task. Previous work proposes learning a diverse set of agents by diversifying the stateaction distribution of the agents. However, without information about the task's goal, the diversified behaviors are not motivated to find other important, albeit nonoptimal, solutions, resulting in only local variations of a solution. In this work, we propose to learn diverse behaviors by looking at policy compatibility while using stateaction information to induce local variations of behaviors. Conceptually, policy compatibility measures whether policies of interest can collectively solve a task. We posit that incompatible policies can be behaviorally different. Based on this idea, we propose a novel objective to learn diverse behaviors. We theoretically show that our novel objective can generate a unique policy, which we incorporate into a populationbased training scheme. Empirically, the proposed method outperforms the baselines in terms of the number of discovered solutions given the same number of agents. 
Rujikorn Charakorn 🔗 
Sat 7:50 a.m.  8:00 a.m.

High Performance Simulation for Scalable MultiAgent Reinforcement Learning
(Spotlight (Contributed))
SlidesLive Video » Multiagent reinforcement learning experiments and opensource training environments are typically limited in scale, supporting tens or sometimes up to hundreds of interacting agents. In this paper we demonstrate the use of Vogue, a high performance agent based modelling framework. Vogue serves as a multiagent training environment, supporting thousands to tens of thousands of interacting agents while maintaining high training throughput by running both the environment and reinforcement learning agents on the GPU. High performance multiagent environments at this scale have the potential to enable the learning of robust and flexible policies for use in agent based models and simulations of complex systems. We demonstrate training performance with two newly developed, large scale multiagent training environments. Moreover, we show that these environments can train shared reinforcement learning policies on timescales of minutes and hours. 
Jordan LanghamLopez 🔗 
Sat 8:00 a.m.  8:20 a.m.

Evology: an EmpiricallyCalibrated Market Ecology AgentBased Model for Trading Strategy Search
(Oral (Contributed))
SlidesLive Video » Market ecology views financial markets as ecosystems of diverse, interacting and evolving trading strategies. We present a heterogeneous, empirically calibrated multiagent market ecology agentbased model. We outline its potential as a valuable and challenging training ground for optimising trading strategies using machine learning algorithms and defining research tasks. 
Aymeric Vie 🔗 
Sat 8:20 a.m.  8:40 a.m.

Exploring social theory integration in agentbased modelling using multiobjective grammatical evolution
(Oral (Contributed))
SlidesLive Video » In Generative Social Science, modellers design agents at the microlevel to generate macrolevel a target social phenomenon. In the Inverse Generative Social Science (iGSS), from a target phenomenon, the goal is to search for possible explanatory model structures. This model discovery process is a promising tool to improve the explanatory capability and theory exploration of computational social science. This paper presents a framework for iGSS and applies Grammatical Evolution to an empiricallycalibrated agentbased model of alcohol use. Results of the model discovery process find many alternative rules for agent behaviours with different tradeoffs. Future work should involve domain experts to evaluate the discovered structures in terms of theoretical credibility and knowledge contribution. 
Tuong Manh Vu 🔗 
Sat 8:40 a.m.  9:00 a.m.

Differentiable agentbased epidemiological modeling for endtoend learning
(Oral (Contributed))
SlidesLive Video » Mechanistic simulators are an indispensable tool for epidemiology to explore the behavior of complex, dynamic infections under varying conditions and navigate uncertain environments. ODEbased models are the dominant paradigm that enable fast simulations and are tractable to gradientbased optimization, but make simplifying assumptions about population homogeneity. Agentbased models (ABMs) are an increasingly popular alternative paradigm that can represent the heterogeneity of contact interactions with granular detail and agency of individual behavior. However, conventional ABM frameworks are not differentiable and present challenges in scalability; due to which it is nontrivial to connect them to auxiliary data sources easily. In this paper we introduce GradABM which is a new scalable, fast and differentiable design for epidemiological ABMs. GradABM runs simulations with millionsize populations in few seconds on commodity hardware and enables fast forward and differentiable inverse simulations. This makes it amenable to be merged with deep neural networks and seamlessly integrate heterogeneous data sources to help with calibration, forecasting and policy evaluation. We demonstrate the efficacy of GradABM via extensive experiments with real COVID19 and influenza datasets. We are optimistic that the current work will bring ABM and AI communities closer together. 
Ayush Chopra 🔗 
Sat 9:00 a.m.  10:30 a.m.

Poster Session and Lunch Break
(Poster Session)

🔗 
Sat 10:30 a.m.  11:00 a.m.

Estimating Policy Functions in Payments Systems Using Reinforcement Learning
(Invited Talk)
SlidesLive Video » Highvalue payments systems (HVPSs) are used to settle transactions between large financial institutions and are considered the core national financial infrastructure. In collaboration with the Bank of Canada, we have been exploring the use of reinforcement learning techniques to understand the behaviour of banks participating in the Canadian HVPS. This understanding could help regulators design policies to ensure the safety and efficiency of these systems. 
Pablo Samuel Castro 🔗 
Sat 11:00 a.m.  11:30 a.m.

Latent state estimation for agentbased models using data assimilation
(Invited Talk)
SlidesLive Video » Scientists have recognized the need to build bottomup models for socioeconomic systems. Such models are often framed as heterogeneous agents interacting in a network following the rules of a dynamical system. However, the available data is often aggregated and incomplete, so even if we have a good model of reality, inferring the state of the individual agents remains a big open challenge. Moreover, these models are usually costly to simulate because one has to compute the individual interactions of all the agents in the system. We present a methodology to infer the latent states of agents embedded in a network when the data available is sparse, noisy, and lowdimensional. The methodology is based on the ensemble Kalman filter extended with a network localization technique that uses the system’s topology to improve the accuracy of the estimations. Our methodology has the following desired properties for bottomup socioeconomic models: i) it treats the model as a black box, so it does not assume any closedform mathematical form of the model a priori, ii) it requires a minimal number of simulations compared to stateoftheart methods, iii) it exploits the underlying topology of the system to improve its predictions, iv) it works for nonlinear systems, v) it is welljustified from a Bayesian perspective, and vi) it is easy to implement. We validate our methodology in two informative examples: 1) a highdimensional approximation of the MackeyGlass chaotic system and 2) the HegselmannKrause bounded confidence (nonlinear) model of opinion dynamics embedded in a social network. While we do not use realworld data to showcase our methodology, we add noise and exogenous shocks to the observations, obtaining accurate predictions in both the observation and the latent state spaces. We aim to help bridge the gap between bottomup modeling and data assimilation techniques in a computationally efficient way. 
Blas Kolic 🔗 
Sat 11:30 a.m.  12:00 p.m.

Physicsinfused learning with ABM
(Invited Talk)
Research at the intersection of physics and machine learning has outsized potential to advance both fields: models and algorithms can be embedded with, or informed by, physics knowledge, and learned models as well as simulators of complex processes and systems can advance experimentation towards understanding. In practice, this physicsML intersection is almost entirely ML surrogates for accelerating an existing numerical simulator. But can we look to ML & AI to discover physics knowledge we don't yet have governing equations for, to recover missing physics and fill gaps in humanexpert understanding? It is this perspective we explore in this talk, in particular the use of agentbased modeling (ABM) as a new abstraction for computational fluid dynamics (CFD), pulling in advanced reinforcement learning (RL) methods from the AI field. We introduce multiagent RL as an automated discovery tool of turbulence and other fluid dynamics models, leveraging the emergent phenomena of ABM to surface the unresolved subgridscale physics. These methods, although nascent, can significantly advance prediction and control of industrial aerodynamics and environmental flows in critical areas like nuclear fusion plasma and atmospheric transport of contaminants. 
Alexander Lavin 🔗 
Sat 12:00 p.m.  12:10 p.m.

Calibrating Agentbased Models to Microdata with Graph Neural Networks
(Spotlight (Contributed))
SlidesLive Video » Calibrating agentbased models to data is among the most fundamental requirements to ensure the model fulfils its desired purpose. In recent years, simulationbased inference methods have emerged as powerful tools for performing this task when the model likelihood function is intractable, as is often the case for agentbased models. In some realworld use cases of agentbased models, both the observed data and the agentbased model output consist of the agents' states and their interactions over time. In such cases, there is a tension between the desire to make full use of the rich information content of such granular data on the one hand, and the need to reduce the dimensionality of the data to prevent difficulties associated with highdimensional learning tasks on the other. A possible resolution is to construct lowerdimensional timeseries through the use of summary statistics describing the macrostate of the system at each time point. However, a poor choice of summary statistics can result in an unacceptable loss of information from the original dataset, dramatically reducing the quality of the resulting calibration. In this work, we instead propose to learn parameter posteriors associated with granular microdata directly using temporal graph neural networks. We will demonstrate that such an approach offers highly compelling inductive biases for Bayesian inference using the raw agentbased model microstates as output. 
Joel Dyer 🔗 
Sat 12:10 p.m.  12:20 p.m.

The StarCraft MultiAgent Challenges+ : Learning of Subtasks and Environmental Benefits without Precise Reward Functions
(Spotlight (Contributed))
SlidesLive Video » Coming Soon 
Mingyu Kim 🔗 
Sat 12:20 p.m.  12:40 p.m.

Estimating the Impact of Coordinated Inauthentic Behavior on Content Recommendations in Social Networks
(Oral (Contributed))
SlidesLive Video » Online disinformation is a dynamic and pervasive problem on social networks as evidenced by a spate of public disasters in light of active efforts to combat it. Since the massive amounts of content generated each day on these platforms is impossible to manually curate, ranking and recommendation algorithms are a key apparatus that drive user interactions. However, the vulnerability of ranking and recommendation algorithms to attack from coordinated campaigns spreading misleading information has been established both theoretically and anecdotally. Unfortunately it is unclear how effective countermeasures to disinformation are in practice due to the limited view we have into the operation of such platforms. We develop a multiagent simulation of a popular social network, Reddit, that aligns with the stateaction space available to real users based on the platform's affordances. We collect millions of realworld interactions from Reddit to estimate the network for each user in our dataset and utilise Reddit's selfdescribed content ranking strategies to compare the impact of coordinated activity on content spread by each algorithm. We expect that this will inform the design of robust content distribution systems that are resilient against targeted attacks by groups of malicious actors. 
Swapneel Mehta 🔗 
Sat 12:40 p.m.  1:00 p.m.

Coffee Break
(Break)

🔗 
Sat 1:00 p.m.  2:00 p.m.

Panel Discussion
SlidesLive Video » 
Christian Schroeder de Witt 🔗 
Sat 2:00 p.m.  2:30 p.m.

Award Ceremony & Closing Remarks
SlidesLive Video » 
Christian Schroeder de Witt 🔗 


LowLoss Subspace Compression for Clean Gains against MultiAgent Backdoor Attacks
(Poster (Contributed))
[
Poster]
SlidesLive Video » Recent exploration of the multiagent backdoor attack demonstrated the backfiring effect, a natural defense against backdoor attacks where backdoored inputs are randomly classified. This yields a sideeffect of low accuracy w.r.t. clean labels, which motivates this paper's work on the construction of multiagent backdoor defenses that maximize accuracy w.r.t. clean labels and minimize that of poison labels. Founded upon agent dynamics and lowloss subspace construction, we contribute three defenses that yield improved multiagent backdoor robustness. 
Siddhartha Datta 🔗 


Deep Learningbased Spatially Explicit Emulation of an AgentBased Simulator for Pandemic in a City
(Poster (Contributed))
[
Poster]
SlidesLive Video » AgentBased Models are very useful for simulation of physical or social processes, such as the spreading of a pandemic in a city. Such models proceed by specifying the behavior of individuals (agents) and their interactions, and parameterizing the process of infection based on such interactions based on the geography and demography of the city. However, such models are computationally very expensive, and the complexity is often linear in the total number of agents. This seriously limits the usage of such models for simulations, which often have to be run hundreds of times for policy planning and even model parameter estimation. An alternative is to develop an emulator, a surrogate model that can predict the AgentBased Simulator's output based on its initial conditions and parameters. In this paper, we discuss a Deep Learning model based on Dilated Convolutional Neural Network that can emulate such an agent based model with high accuracy. We show that use of this model instead of the original AgentBased Model provides us major gains in the speed of simulations, allowing much quicker calibration to observations, and more extensive scenario analysis. The models we consider are spatially explicit, as the locations of the infected individuals are simulated instead of the gross counts. Another aspect of our emulation framework is its divideandconquer approach that divides the city into several small overlapping blocks and carries out the emulation in them parallelly, after which these results are merged together. This ensures that the same emulator can work for a city of any size, and also provides significant improvement of time complexity of the emulator, compared to the original simulator. 
Varun Madhavan 🔗 


Convergence and Price of Anarchy Guarantees of the Softmax Policy Gradient in Markov Potential Games
(Poster (Contributed))
[
Poster]
SlidesLive Video » 
Dingyang Chen 🔗 


A Variational Approach to Mutual InformationBased Coordination for MultiAgent Reinforcement Learning
(Poster (Contributed))
SlidesLive Video » 
WOOJUN KIM 🔗 


Risk Perspective Exploration in Distributional Reinforcement Learning
(Poster (Contributed))
[
Poster]
SlidesLive Video » Distributional reinforcement learning shows stateoftheart performance in continuous and discrete control settings with the properties of variance and risk, which can be utilized as a means of exploration. However, the exploration method using risk property is hard to find, while various exploration methods use the variance of return distribution per action in Distributional RL. In this paper, we propose risk scheduling methods, a risk perspective exploration, that explore risk levels and optimistic actions. We show the performance improvement of DMIX algorithm by risk scheduling in a multiagent setting with various experiments. 
Jihwan Oh 🔗 


The AI Macroeconomy: A New Set of Benchmarks for Multiagent Reinforcement Learning Models
(Poster (Contributed))
[
Poster]
SlidesLive Video » We propose a new set of challenging benchmark gym environments for testing single and multiagent reinforcement learning environments. Singleagent environments are based on a simple consumptionsaving decision problem. In each period, agents face an exogenous positive draw that represents how much income they will have in this period. In response, agents may choose what fraction of that income they would like to consume immediately for a reward, or save and get a return going forward on it. In the full version of the problem, all agents' saving decisions generate a price via market clearing. Agents then must learn what their value will be conditioned on the current state. This environment will provide a challenging, potentially nonstationary environment where agents' actions have critical effects on other agents' actions, albeit via a common observation. This environment will be made publicly available via a Github repository and opensource. 
Brandon Kaplowitz 🔗 


The Robustness of Inverse Reinforcement Learning
(Poster (Contributed))
Reinforcement learning research experienced substantial jumps in its progress after the first achievement on utilizing deep neural networks to approximate the stateaction value function in highdimensional states. While deep reinforcement learning algorithms are currently being employed in many different tasks from industrial control to biomedical applications, yet the fact that an MDP has to provide a clear reward function limits the tasks that can be achieved via reinforcement learning. In this line of research, some studies proposed to directly learn a policy from observing expert trajectories (i.e. imitation learning), and others proposed to learn a reward function from the expert demonstrations (i.e. inverse reinforcement learning). In this paper we will focus on robustness and vulnerabilities of deep imitation learning and deep inverse reinforcement learning policies. Furthermore, we will layout nonrobust features learnt by the deep inverse reinforcement learning policies. We conduct experiments in the Arcade Learning Environment (ALE), and compare the nonrobust features learnt by the deep inverse reinforcement learning algorithms to vanilla trained deep reinforcement learning policies. We hope that our study can provide a basis for the future discussions on the robustness of both deep inverse reinforcement learning and deep reinforcement learning. 
Ezgi Korkmaz 🔗 


Introducing our mentee Aymeric Vie
(Mentorship Program Presentation)
SlidesLive Video » 
Aymeric Vie 🔗 


Introducing our mentor Yongli Zhu
(Mentorship Program Presentation)
SlidesLive Video » 
Yongli Zhu 🔗 


The StarCraft MultiAgent Challenges+ : Learning of Subtasks and Environmental Benefits without Precise Reward Functions (Poster)
(Poster (Contributed))
[
Poster]

Mingyu Kim 🔗 


Calibrating Agentbased Models to Microdata with Graph Neural Networks
(Poster (Contributed))
[
Poster]
Calibrating agentbased models to data is among the most fundamental requirements to ensure the model fulfils its desired purpose. In recent years, simulationbased inference methods have emerged as powerful tools for performing this task when the model likelihood function is intractable, as is often the case for agentbased models. In some realworld use cases of agentbased models, both the observed data and the agentbased model output consist of the agents' states and their interactions over time. In such cases, there is a tension between the desire to make full use of the rich information content of such granular data on the one hand, and the need to reduce the dimensionality of the data to prevent difficulties associated with highdimensional learning tasks on the other. A possible resolution is to construct lowerdimensional timeseries through the use of summary statistics describing the macrostate of the system at each time point. However, a poor choice of summary statistics can result in an unacceptable loss of information from the original dataset, dramatically reducing the quality of the resulting calibration. In this work, we instead propose to learn parameter posteriors associated with granular microdata directly using temporal graph neural networks. We will demonstrate that such an approach offers highly compelling inductive biases for Bayesian inference using the raw agentbased model microstates as output. 
Joel Dyer 🔗 
Author Information
Christian Schroeder (University of Oxford)
Yang Zhang (Banque Canada)
Anisoara Calinescu (University of Oxford)
Dylan Radovic (McKinsey)
Prateek Gupta (University of Oxford)
Jakob Foerster (University of Oxford)
More from the Same Authors

2022 : Fixed Points in Cyber Space: Rethinking Optimal Evasion Attacks in the Age of AINIDS »
Christian Schroeder · Yongchao Huang · Phil Torr · Martin Strohmeier 
2022 : Fixed Points in Cyber Space: Rethinking Optimal Evasion Attacks in the Age of AINIDS »
Christian Schroeder · Yongchao Huang · Phil Torr · Martin Strohmeier 
2022 Poster: Communicating via Markov Decision Processes »
Samuel Sokota · Christian Schroeder · Maximilian Igl · Luisa Zintgraf · Phil Torr · Martin Strohmeier · Zico Kolter · Shimon Whiteson · Jakob Foerster 
2022 Spotlight: Communicating via Markov Decision Processes »
Samuel Sokota · Christian Schroeder · Maximilian Igl · Luisa Zintgraf · Phil Torr · Martin Strohmeier · Zico Kolter · Shimon Whiteson · Jakob Foerster 
2021 Poster: Randomized Entitywise Factorization for MultiAgent Reinforcement Learning »
Shariq Iqbal · Christian Schroeder · Bei Peng · Wendelin Boehmer · Shimon Whiteson · Fei Sha 
2021 Oral: Randomized Entitywise Factorization for MultiAgent Reinforcement Learning »
Shariq Iqbal · Christian Schroeder · Bei Peng · Wendelin Boehmer · Shimon Whiteson · Fei Sha 
2020 Poster: Revisiting Training Strategies and Generalization Performance in Deep Metric Learning »
Karsten Roth · Timo Milbich · Samrath Sinha · Prateek Gupta · Bjorn Ommer · Joseph Paul Cohen 
2019 : "Ideas" minispotlights »
Kevin McCloskey · Nikola MilojevicDupont · Jonathan Binas · Christian Schroeder · Sasha Luccioni 
2019 : Networking Lunch (provided) + Poster Session »
Abraham Stanway · Alex Robson · Aneesh Rangnekar · Ashesh Chattopadhyay · Ashley Pilipiszyn · Benjamin LeRoy · Bolong Cheng · Ce Zhang · Chaopeng Shen · Christian Schroeder · Christian Clough · Clement DUHART · Clement Fung · Cozmin Ududec · Dali Wang · David Dao · di wu · Dimitrios Giannakis · Dino Sejdinovic · Doina Precup · Duncan WatsonParris · Gege Wen · George Chen · Gopal Erinjippurath · Haifeng Li · Han Zou · Herke van Hoof · Hillary A Scannell · Hiroshi Mamitsuka · Hongbao Zhang · Jaegul Choo · James Wang · James Requeima · Jessica Hwang · Jinfan Xu · Johan Mathe · Jonathan Binas · Joonseok Lee · Kalai Ramea · Kate Duffy · Kevin McCloskey · Kris Sankaran · Lester Mackey · Letif Mones · Loubna Benabbou · Lynn Kaack · Matthew Hoffman · Mayur Mudigonda · Mehrdad Mahdavi · Michael McCourt · Mingchao Jiang · Mohammad Mahdi Kamani · Neel Guha · Niccolo Dalmasso · Nick Pawlowski · Nikola MilojevicDupont · Paulo Orenstein · Pedram Hassanzadeh · Pekka Marttinen · Ramesh Nair · Sadegh Farhang · Samuel Kaski · Sandeep Manjanna · Sasha Luccioni · Shuby Deshpande · Soo Kim · Soukayna Mouatadid · Sunghyun Park · Tao Lin · Telmo Felgueira · Thomas Hornigold · Tianle Yuan · Tom Beucler · Tracy Cui · Volodymyr Kuleshov · Wei Yu · yang song · Ydo Wexler · Yoshua Bengio · Zhecheng Wang · Zhuangfang Yi · Zouheir Malki 
2018 Poster: QMIX: Monotonic Value Function Factorisation for Deep MultiAgent Reinforcement Learning »
Tabish Rashid · Mikayel Samvelyan · Christian Schroeder · Gregory Farquhar · Jakob Foerster · Shimon Whiteson 
2018 Oral: QMIX: Monotonic Value Function Factorisation for Deep MultiAgent Reinforcement Learning »
Tabish Rashid · Mikayel Samvelyan · Christian Schroeder · Gregory Farquhar · Jakob Foerster · Shimon Whiteson