Value Aggregation with Uncertainty in Online Decentralized MARL

Multi-agent reinforcement learning (MARL) has received increasing attention for solving complex decision-making tasks. Networked MARL approaches offer a decentralized solution for parameter sharing to accelerate training via value aggregation. However, existing federated aggregations rely on convex averaging that may fail to converge to global optima and cause learning rollback in the online learning setting. In this paper, we formally characterize the learning rollback phenomenon arising from aggregating value estimates with unequal uncertainty under heterogeneous online update depths. We propose a novel adaptive global consensus (AGC) mechanism for Q-value aggregation in decentralized MARL policy evaluation, which dynamically adjusts aggregation weights based on agents’ uncertainty. We establish theoretical guarantees on accelerated convergence and bounded learning variance with empirical validations, advancing the state-of-art MARL theory.

The philosophical question, ''Are More Heads better than Fewer but Smarter?'', naturally arises when constructing policy aggregation strategies in online multi-agent learning systems. We find that the answer is not always yes, due to the existence of learning rollback phenomenon in online settings, which degrades the system learning performance. This is also the answer to why aggregated parameters do not always form an effective agent consensus. To answer our research question--'How can we design an efficient online algorithm to bypass learning rollback with uncertainty in networked dec-MARL?'--we first formally studied the observed learning rollback issue. Based on the analysis, we propose the FedAGC framework, which designs the adaptive aggregation weights w.r.t. the quantified parameter uncertainty whilst ensuring theoretical learning guarantees. As one of the first paper in decentralized networked MARL theory, our FedAGC framework bridges the weight design with parameter uncertainty. This can be further extended to stochastic learning processes in more general NN-based function approximations in order to accelerate and understand multi-agent learning.