Effectively training a robust agent that can cooperate with unseen agents requires diverse training partner agents. Nonetheless, obtaining cooperative agents with diverse behaviors is a challenging task. Previous work proposes learning a diverse set of agents by diversifying the state-action distribution of the agents. However, without information about the task's goal, the diversified behaviors are not motivated to find other important, albeit non-optimal, solutions, resulting in only local variations of a solution. In this work, we propose to learn diverse behaviors by looking at policy compatibility while using state-action information to induce local variations of behaviors. Conceptually, policy compatibility measures whether policies of interest can collectively solve a task. We posit that incompatible policies can be behaviorally different. Based on this idea, we propose a novel objective to learn diverse behaviors. We theoretically show that our novel objective can generate a unique policy, which we incorporate into a population-based training scheme. Empirically, the proposed method outperforms the baselines in terms of the number of discovered solutions given the same number of agents.

Multi-agent reinforcement learning experiments and open-source training environments are typically limited in scale, supporting tens or sometimes up to hundreds of interacting agents. In this paper we demonstrate the use of Vogue, a high performance agent based modelling framework. Vogue serves as a multi-agent training environment, supporting thousands to tens of thousands of interacting agents while maintaining high training throughput by running both the environment and reinforcement learning agents on the GPU. High performance multi-agent environments at this scale have the potential to enable the learning of robust and flexible policies for use in agent based models and simulations of complex systems. We demonstrate training performance with two newly developed, large scale multi-agent training environments. Moreover, we show that these environments can train shared reinforcement learning policies on time-scales of minutes and hours.

Market ecology views financial markets as ecosystems of diverse, interacting and evolving trading strategies. We present a heterogeneous, empirically calibrated multi-agent market ecology agent-based model. We outline its potential as a valuable and challenging training ground for optimising trading strategies using machine learning algorithms and defining research tasks.

In Generative Social Science, modellers design agents at the micro-level to generate macro-level a target social phenomenon. In the Inverse Generative Social Science (iGSS), from a target phenomenon, the goal is to search for possible explanatory model structures. This model discovery process is a promising tool to improve the explanatory capability and theory exploration of computational social science. This paper presents a framework for iGSS and applies Grammatical Evolution to an empirically-calibrated agent-based model of alcohol use. Results of the model discovery process find many alternative rules for agent behaviours with different trade-offs. Future work should involve domain experts to evaluate the discovered structures in terms of theoretical credibility and knowledge contribution.

Mechanistic simulators are an indispensable tool for epidemiology to explore the behavior of complex, dynamic infections under varying conditions and navigate uncertain environments. ODE-based models are the dominant paradigm that enable fast simulations and are tractable to gradient-based optimization, but make simplifying assumptions about population homogeneity. Agent-based models (ABMs) are an increasingly popular alternative paradigm that can represent the heterogeneity of contact interactions with granular detail and agency of individual behavior. However, conventional ABM frameworks are not differentiable and present challenges in scalability; due to which it is non-trivial to connect them to auxiliary data sources easily. In this paper we introduce GradABM which is a new scalable, fast and differentiable design for epidemiological ABMs. GradABM runs simulations with million-size populations in few seconds on commodity hardware and enables fast forward and differentiable inverse simulations. This makes it amenable to be merged with deep neural networks and seamlessly integrate heterogeneous data sources to help with calibration, forecasting and policy evaluation. We demonstrate the efficacy of GradABM via extensive experiments with real COVID-19 and influenza datasets. We are optimistic that the current work will bring ABM and AI communities closer together.

High-value payments systems (HVPSs) are used to settle transactions between large financial institutions and are considered the core national financial infrastructure. In collaboration with the Bank of Canada, we have been exploring the use of reinforcement learning techniques to understand the behaviour of banks participating in the Canadian HVPS. This understanding could help regulators design policies to ensure the safety and efficiency of these systems.

Scientists have recognized the need to build bottom-up models for socio-economic systems. Such models are often framed as heterogeneous agents interacting in a network following the rules of a dynamical system. However, the available data is often aggregated and incomplete, so even if we have a good model of reality, inferring the state of the individual agents remains a big open challenge. Moreover, these models are usually costly to simulate because one has to compute the individual interactions of all the agents in the system. We present a methodology to infer the latent states of agents embedded in a network when the data available is sparse, noisy, and low-dimensional. The methodology is based on the ensemble Kalman filter extended with a network localization technique that uses the system’s topology to improve the accuracy of the estimations. Our methodology has the following desired properties for bottom-up socioeconomic models: i) it treats the model as a black box, so it does not assume any closed-form mathematical form of the model a priori, ii) it requires a minimal number of simulations compared to state-of-the-art methods, iii) it exploits the underlying topology of the system to improve its predictions, iv) it works for nonlinear systems, v) it is well-justified from a Bayesian perspective, and vi) it is easy to implement. We validate our methodology in two informative examples: 1) a high-dimensional approximation of the Mackey-Glass chaotic system and 2) the Hegselmann-Krause bounded confidence (nonlinear) model of opinion dynamics embedded in a social network. While we do not use real-world data to showcase our methodology, we add noise and exogenous shocks to the observations, obtaining accurate predictions in both the observation and the latent state spaces. We aim to help bridge the gap between bottom-up modeling and data assimilation techniques in a computationally efficient way.

Research at the intersection of physics and machine learning has outsized potential to advance both fields: models and algorithms can be embedded with, or informed by, physics knowledge, and learned models as well as simulators of complex processes and systems can advance experimentation towards understanding. In practice, this physics-ML intersection is almost entirely ML surrogates for accelerating an existing numerical simulator. But can we look to ML & AI to discover physics knowledge we don't yet have governing equations for, to recover missing physics and fill gaps in human-expert understanding? It is this perspective we explore in this talk, in particular the use of agent-based modeling (ABM) as a new abstraction for computational fluid dynamics (CFD), pulling in advanced reinforcement learning (RL) methods from the AI field. We introduce multi-agent RL as an automated discovery tool of turbulence and other fluid dynamics models, leveraging the emergent phenomena of ABM to surface the unresolved subgrid-scale physics. These methods, although nascent, can significantly advance prediction and control of industrial aerodynamics and environmental flows in critical areas like nuclear fusion plasma and atmospheric transport of contaminants.

Calibrating agent-based models to data is among the most fundamental requirements to ensure the model fulfils its desired purpose. In recent years, simulation-based inference methods have emerged as powerful tools for performing this task when the model likelihood function is intractable, as is often the case for agent-based models. In some real-world use cases of agent-based models, both the observed data and the agent-based model output consist of the agents' states and their interactions over time. In such cases, there is a tension between the desire to make full use of the rich information content of such granular data on the one hand, and the need to reduce the dimensionality of the data to prevent difficulties associated with high-dimensional learning tasks on the other. A possible resolution is to construct lower-dimensional time-series through the use of summary statistics describing the macrostate of the system at each time point. However, a poor choice of summary statistics can result in an unacceptable loss of information from the original dataset, dramatically reducing the quality of the resulting calibration. In this work, we instead propose to learn parameter posteriors associated with granular microdata directly using temporal graph neural networks. We will demonstrate that such an approach offers highly compelling inductive biases for Bayesian inference using the raw agent-based model microstates as output.

Coming Soon

Online disinformation is a dynamic and pervasive problem on social networks as evidenced by a spate of public disasters in light of active efforts to combat it. Since the massive amounts of content generated each day on these platforms is impossible to manually curate, ranking and recommendation algorithms are a key apparatus that drive user interactions. However, the vulnerability of ranking and recommendation algorithms to attack from coordinated campaigns spreading misleading information has been established both theoretically and anecdotally. Unfortunately it is unclear how effective countermeasures to disinformation are in practice due to the limited view we have into the operation of such platforms. We develop a multiagent simulation of a popular social network, Reddit, that aligns with the state-action space available to real users based on the platform's affordances. We collect millions of real-world interactions from Reddit to estimate the network for each user in our dataset and utilise Reddit's self-described content ranking strategies to compare the impact of coordinated activity on content spread by each algorithm. We expect that this will inform the design of robust content distribution systems that are resilient against targeted attacks by groups of malicious actors.

Recent exploration of the multi-agent backdoor attack demonstrated the backfiring effect, a natural defense against backdoor attacks where backdoored inputs are randomly classified. This yields a side-effect of low accuracy w.r.t. clean labels, which motivates this paper's work on the construction of multi-agent backdoor defenses that maximize accuracy w.r.t. clean labels and minimize that of poison labels. Founded upon agent dynamics and low-loss subspace construction, we contribute three defenses that yield improved multi-agent backdoor robustness.

Agent-Based Models are very useful for simulation of physical or social processes, such as the spreading of a pandemic in a city. Such models proceed by specifying the behavior of individuals (agents) and their interactions, and parameterizing the process of infection based on such interactions based on the geography and demography of the city. However, such models are computationally very expensive, and the complexity is often linear in the total number of agents. This seriously limits the usage of such models for simulations, which often have to be run hundreds of times for policy planning and even model parameter estimation. An alternative is to develop an emulator, a surrogate model that can predict the Agent-Based Simulator's output based on its initial conditions and parameters. In this paper, we discuss a Deep Learning model based on Dilated Convolutional Neural Network that can emulate such an agent based model with high accuracy. We show that use of this model instead of the original Agent-Based Model provides us major gains in the speed of simulations, allowing much quicker calibration to observations, and more extensive scenario analysis. The models we consider are spatially explicit, as the locations of the infected individuals are simulated instead of the gross counts. Another aspect of our emulation framework is its divide-and-conquer approach that divides the city into several small overlapping blocks and carries out the emulation in them parallelly, after which these results are merged together. This ensures that the same emulator can work for a city of any size, and also provides significant improvement of time complexity of the emulator, compared to the original simulator.

Distributional reinforcement learning shows state-of-the-art performance in continuous and discrete control settings with the properties of variance and risk, which can be utilized as a means of exploration. However, the exploration method using risk property is hard to find, while various exploration methods use the variance of return distribution per action in Distributional RL. In this paper, we propose risk scheduling methods, a risk perspective exploration, that explore risk levels and optimistic actions. We show the performance improvement of DMIX algorithm by risk scheduling in a multi-agent setting with various experiments.

We propose a new set of challenging benchmark gym environments for testing single- and multi-agent reinforcement learning environments. Single-agent environments are based on a simple consumption-saving decision problem. In each period, agents face an exogenous positive draw that represents how much income they will have in this period. In response, agents may choose what fraction of that income they would like to consume immediately for a reward, or save and get a return going forward on it. In the full version of the problem, all agents' saving decisions generate a price via market clearing. Agents then must learn what their value will be conditioned on the current state. This environment will provide a challenging, potentially non-stationary environment where agents' actions have critical effects on other agents' actions, albeit via a common observation. This environment will be made publicly available via a Github repository and open-source.

Reinforcement learning research experienced substantial jumps in its progress after the first achievement on utilizing deep neural networks to approximate the state-action value function in high-dimensional states. While deep reinforcement learning algorithms are currently being employed in many different tasks from industrial control to biomedical applications, yet the fact that an MDP has to provide a clear reward function limits the tasks that can be achieved via reinforcement learning. In this line of research, some studies proposed to directly learn a policy from observing expert trajectories (i.e. imitation learning), and others proposed to learn a reward function from the expert demonstrations (i.e. inverse reinforcement learning). In this paper we will focus on robustness and vulnerabilities of deep imitation learning and deep inverse reinforcement learning policies. Furthermore, we will layout non-robust features learnt by the deep inverse reinforcement learning policies. We conduct experiments in the Arcade Learning Environment (ALE), and compare the non-robust features learnt by the deep inverse reinforcement learning algorithms to vanilla trained deep reinforcement learning policies. We hope that our study can provide a basis for the future discussions on the robustness of both deep inverse reinforcement learning and deep reinforcement learning.

Calibrating agent-based models to data is among the most fundamental requirements to ensure the model fulfils its desired purpose. In recent years, simulation-based inference methods have emerged as powerful tools for performing this task when the model likelihood function is intractable, as is often the case for agent-based models. In some real-world use cases of agent-based models, both the observed data and the agent-based model output consist of the agents' states and their interactions over time. In such cases, there is a tension between the desire to make full use of the rich information content of such granular data on the one hand, and the need to reduce the dimensionality of the data to prevent difficulties associated with high-dimensional learning tasks on the other. A possible resolution is to construct lower-dimensional time-series through the use of summary statistics describing the macrostate of the system at each time point. However, a poor choice of summary statistics can result in an unacceptable loss of information from the original dataset, dramatically reducing the quality of the resulting calibration. In this work, we instead propose to learn parameter posteriors associated with granular microdata directly using temporal graph neural networks. We will demonstrate that such an approach offers highly compelling inductive biases for Bayesian inference using the raw agent-based model microstates as output.