Towards Complete Multi-Agent Coordination Policy Learning via Denoising Maximum Entropy Optimization

Parameter sharing is a widely used technique in Multi-Agent Reinforcement Learning (MARL) that enhances sample efficiency by equipping agents with a unified policy. While effective in homogeneous settings, it often struggles in heterogeneous environments where agents possess diverse capabilities. Conversely, learning customized policies for agents can resolve knowledge conflicts but significantly hinders knowledge transfer, thereby reducing learning efficiency. Existing approaches attempt to balance this trade-off using clustering or agent-specific masks, but they typically rely on strong environment-specific priors and struggle in settings where the team exhibits multi-modal policies. To address these limitations, we propose Dspic, an efficient shared-policy algorithm grounded in the maximum entropy framework. Specifically, Dspic employs self-supervised learning to extract discriminative role embeddings for each agent. These embeddings guide a complete division of the observation space, providing a theoretical guarantee for the optimality of parameter sharing. Furthermore, to handle the increased observation complexity and diversity resulting from this division, Dspic incorporates a diffusion policy, enhancing the capacity to model complex action distributions while enabling efficient learning. Extensive experiments on MaMuJoCo, SMAC, SMACv2, and LBF demonstrate that Dspic achieves superior sample efficiency while maintaining asymptotic optimality.

In a team, members often have different roles, meaning they should behave differently when faced with the same observations. Some existing multi-agent strategies allow these members to share a policy network, which clearly violates this characteristic and theoretically leads to suboptimality. Directly customizing independent policies for each member often performs poorly in terms of sample efficiency. While some methods attempt to address this issue, they consistently underperform or exhibit instability in complex environments. To solve this problem, we embed role semantic information into a matrix space, which can be used to spatially partition the agent's input vectors through projection. Through orthogonal constraints, our method makes the input spaces of different agents approach disjoint, achieving excellent results in practice, demonstrating scalability and stability.