Graph Diffusion for Robust Multi-Agent Coordination

Offline multi-agent reinforcement learning (MARL) struggles to estimate out-of-distribution states and actions due to the absence of real-time environmental feedback. While diffusion models show promise in addressing these challenges, their application primarily focuses on independently diffusing the historical trajectories of individual agents, neglecting crucial multi-agent coordination dynamics and reducing policy robustness in dynamic environments. In this paper, we propose MCGD, a novel Multi-agent Coordination framework based on Graph Diffusion models to improve the effectiveness and robustness of collaborative policies. Specifically, we begin by constructing a sparse coordination graph that includes continuous node attributes and discrete edge attributes to effectively identify the underlying dynamics of multi-agent interactions. Next, we derive transition probabilities between edge categories and present adaptive categorical diffusion to capture the structure diversity of multi-agent coordination. Leveraging this coordination structure, we define neighbor-dependent forward noise and develop anisotropic diffusion to enhance the action diversity of each agent. Extensive experiments across various multi-agent environments demonstrate that MCGD significantly outperforms existing state-of-the-art baselines in coordination performance and policy robustness in dynamic environments.

Training multiple agents to cooperate effectively in complex environments—known as multi-agent reinforcement learning (MARL)—is difficult when there is no real-time feedback from the environment. Traditional methods often fail when faced with unfamiliar situations, partly because they ignore the essential coordination between agents. Recent diffusion models have shown potential but mostly focus on each agent individually, missing the critical group dynamics.In our work, we introduce MCGD, a novel framework that uses graph diffusion models to better capture and enhance multi-agent coordination. We first represent the agents and their interactions in a coordination graph that accounts for both the agents’ continuous states and the discrete relationships between them. We then propose a new method called adaptive categorical diffusion to understand the changing interaction patterns among agents. Additionally, we develop an anisotropic diffusion process to improve the variety and robustness of each agent’s actions.Our extensive experiments show that MCGD leads to superior coordination and more reliable decision-making across multiple test environments, outperforming current state-of-the-art methods.