Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policies

Online Multi-Agent Reinforcement Learning (MARL) is a prominent framework for efficient agent coordination. Crucially, enhancing policy expressiveness is pivotal for achieving superior performance. Diffusion-based generative models are well-positioned to meet this demand, having demonstrated remarkable expressiveness and multimodal representation in image generation and offline settings. Yet, their potential in online MARL remains largely under-explored. A major obstacle is that the intractable likelihoods of diffusion models impede entropy-based exploration and coordination. To tackle this challenge, we propose among the first Online off-policy MARL framework using Diffusion policies (**OMAD**) to orchestrate coordination. Our key innovation is a relaxed policy objective that maximizes scaled joint entropy, facilitating effective exploration without relying on tractable likelihood. Complementing this, within the centralized training with decentralized execution (CTDE) paradigm, we employ a joint distributional value function to optimize decentralized diffusion policies. It leverages tractable entropy-augmented targets to guide the simultaneous updates of diffusion policies, thereby ensuring stable coordination. Extensive evaluations on MPE and MAMuJoCo establish our method as the new state-of-the-art across $10$ diverse tasks, demonstrating a remarkable $2.5\times$ to $5\times$ improvement in sample efficiency.

(1) Problem To enable multiple AIs, such as autonomous vehicle fleets or search-and-rescue robots, to collaborate seamlessly, they require extremely flexible "brains." While diffusion models, the technology behind AI image generation, are remarkably powerful and can provide the necessary flexibility, their underlying complex mathematical mechanisms cause AIs to easily get "stuck" when exploring new strategies. For this reason, this powerful technology has previously been difficult to apply to real-time guidance of AI team collaboration. (2) Solution To untangle this mathematical deadlock, we developed a new framework called OMAD, the first successful method to apply diffusion models to online training of AI teams. We cleverly provided the AIs with a set of "relaxed requirement" training rules, allowing them to explore boldly and maintain coordination even when they cannot compute exact mathematical probabilities. Under this framework, AIs receive centralized guidance during training as a collective, yet each AI can independently make smart and coordinated decisions during task execution. (3) Impact This breakthrough delivers remarkable results. Across ten complex team task benchmarks, OMAD outperformed all existing methods, setting entirely new state-of-the-art records. More importantly, it dramatically accelerated AI team learning speed by 2.5 to 5 times. This research unlocks the potential of generative AI in multi-agent systems, paving the way for more efficient robot team deployments in the real world.