IEC: When Information-Driven Exploration Meets Spectral Consensus via Primal–Dual Reward Regularization in Decentralized Multi-Agent RL

Decentralized multi-agent reinforcement learning faces a persistent exploration–coordination tension: intrinsic rewards promote exploration under sparse feedback, yet effective cooperation requires agents’ behaviors to remain consistent over a limited communication graph. Existing methods often combine exploration bonuses and coordination regularizers with fixed-weight schedules, making them hard to tune and prone to either fragmented conventions or premature behavioral collapse. We propose the IEC (Isomorphic Exploration-Consensus) framework that couples exploration and coordination through a single constrained objective: maximize task return augmented with two complementary exploration signals, dynamics-based information gain and state-coverage novelty, while constraining graph-induced policy disagreement via a spectral smoothness penalty on neighboring agents, which can be interpreted as a Dirichlet-energy regularizer on the communication graph. IEC optimizes the resulting Lagrangian with a lightweight primal–dual update that adapts the consensus multiplier from observed constraint violations, yielding an automatic shift from diverse exploration to stable cooperative conventions. Across three distinct benchmarks, IEC achieves superior performance.

Many real-world AI systems involve several decision-making agents working together, such as robots, drones, traffic-control units, or game characters. These agents often need to learn by trial and error while only sharing limited information with nearby teammates. This creates a difficult balance: each agent must explore new actions to discover better strategies, but the team must also stay coordinated so that individual actions do not conflict with one another. This paper proposes IEC, a learning framework that helps a team of agents balance exploration and coordination automatically. Instead of using fixed hand-designed weights to decide how much agents should explore or align with teammates, IEC adjusts this balance during learning. When agents behave too differently from their neighbors, the method increases the pressure for coordination. When the team is already well aligned, it allows more freedom for exploration. We evaluate IEC in three types of cooperative tasks. Across these tasks, IEC helps agents discover useful strategies more reliably while maintaining coordinated behavior. The results suggest that adaptive balancing between exploration and teamwork can make multi-agent systems more robust, especially when rewards are sparse and communication is limited.