HPS: Hyperspherical Parameter Sharing for Efficient Multi-Agent Reinforcement Learning

Parameter Sharing (PS) is widely used to improve efficiency in Multi-Agent Reinforcement Learning (MARL), but it can limit behavioral diversity and degrade performance. This limitation stems from gradient conflicts among agents on shared weights, which hinders effective policy learning. To fully characterize this phenomenon, we propose Geometric Gradient Decomposition Analysis that decomposes gradients with respect to weight vector into radial (scale) and tangential (direction) components and uncover a key insight: agents largely agree on directional updates but substantially disagree on scale updates. Consequently, while recent methods split the shared network into agent-specific subnetworks to mitigate conflicts, they also discard shared directional updates, limiting training efficiency. To address this issue, we propose Hyperspherical Parameter Sharing (HPS), which explicitly decouples direction and scale in parameter sharing. Specifically, HPS constrains the shared backbone weights onto a Riemannian manifold(unit hypersphere), enforcing purely directional learning. Building on this, an agent-specific scale generator outputs multiplicative modulation factors to adjust each agent’s scales, thus preserving heterogeneous response magnitudes without disrupting the shared directions. Experiments on SMAC, SMACv2, VMAS and Predator Prey demonstrate that HPS effectively resolves the scale conflict, significantly outperforming state-of-the-art methods.

Modern AI often needs groups of agents to work together, such as robots, vehicles, or game characters. When all agents use exactly the same neural network, training is efficient, but agents with different roles can become too similar and make poor decisions. We found that agents usually agree on which features should guide their decisions, but disagree on how strongly those features should influence each agent. Based on this observation, our method, Hyperspherical Parameter Sharing (HPS), separates these two parts of learning: a shared model learns common decision directions, while a lightweight agent-specific module adjusts each agent’s response strength. This lets agents keep the benefits of sharing knowledge without forcing them to behave identically. In cooperative benchmarks including StarCraft micromanagement and physics-based multi-agent tasks, HPS helped teams learn better strategies than existing methods that make agents share a model. The result is a more efficient way to train cooperative AI systems in which teammates must coordinate while still performing different roles. Such methods could support future applications in robotics, traffic control, and other settings where many decision-making agents must work together reliably.