Offline Multi-Agent Reinforcement Learning via Sequential Score Decomposition

Offline cooperative multi-agent reinforcement learning (MARL) faces unique challenges due to the distribution shift between online and offline data collection. While online MARL typically converges to a single coordinated joint policy, offline datasets are often mixtures of diverse cooperative behaviors, resulting in highly multimodal joint behavior distributions. In such settings, independent policy regularization often misaligns joint policy constraints and leads to severe distribution shift. To address this, we propose OMSD, which sequentially decomposes the joint behavior policy into individual conditional distributions and leverages diffusion-based generative models to provide modality-coordinated regularization for each agent. Combined with centralized critic guidance, OMSD achieves coordinated exploration within high-value, in-distribution regions, and avoids out-of-distribution joint actions. Experiments across multiple datasets on various continuous control tasks demonstrate that OMSD consistently achieves state-of-the-art performance, especially in challenging multimodal scenarios. Our results highlight the necessity of modality-aware coordination for robust offline MARL.

Many real-world tasks require several decision-making agents to work together, such as robots coordinating their movements or autonomous systems sharing a common goal. A major challenge is learning from previously collected data, because such data may contain many different ways to solve the same task. If each agent learns from only its own part of the data, the agents may combine pieces of different strategies and end up with actions that were never seen together in the original data. This paper proposes OMSD, a method that helps agents learn coordinated behavior from offline data. Instead of treating each agent separately, OMSD learns how one agent’s action should depend on the actions of earlier agents during training. It uses this information to guide each agent toward actions that are both useful and consistent with the data. At test time, the agents can still act independently and efficiently. Experiments on several multi-agent control tasks show that OMSD learns stronger coordinated policies than previous methods, especially when the offline data contains diverse or low-quality behavior.