DecFus: Decentralized Layer-wise Fusion with Dynamic Exploration and Exploitation

Decentralized Federated Learning (DFL) enables collaborative model training across connected clients without a central server, effectively mitigating communication bottlenecks and avoiding the single point of failure in Centralized Federated Learning (CFL). However, existing DFL methods mostly focus on parameter averaging with compromised update directions, which limits their performance potential due to insufficient exploration of the loss landscape, especially for complex models. We observe that layer exchanges among clients enhance exploration while introducing instability due to highly diverse update directions. To address these limitations, we propose Decentralized Layer-wise Fusion (DecFus), the first DFL framework that unifies layer-level exchange and averaging to balance exploration and exploitation. DecFus dynamically transitions the decentralized training process from exploration-dominant to exploitation-dominant phases, guided by the loss variance among connected neighbors. Furthermore, a layer-wise fusion strategy, informed by pairwise cosine similarity, categorizes all layers into two groups: an exchange group for exploration and an averaging group for exploitation. Specifically, we theoretically establish the convergence of DecFus without relying on the common assumption in existing literature that the aggregation matrix must be doubly stochastic. Extensive experiments demonstrate that DecFus achieves superior performance in both IID and non-IID scenarios, substantially outperforming existing CFL and DFL methods.

Decentralized federated learning allows connected clients to train machine learning models collaboratively without relying on a central server. This can reduce communication bottlenecks and avoid the risk of a single point of failure. However, existing methods mainly rely on parameter averaging, which may limit model performance because it does not sufficiently explore the loss landscape, especially for complex models. This paper proposes DecFus, a decentralized layer-wise fusion framework that combines layer exchange and layer averaging. Layer exchange helps improve exploration, while layer averaging supports stable training. DecFus dynamically shifts the training process from exploration-dominant to exploitation-dominant phases, guided by the loss variance among connected clients. It also uses layer-wise similarity to decide which layers should be exchanged and which should be averaged. The paper further provides a convergence analysis without relying on the common doubly stochastic aggregation matrix assumption. Experiments on multiple datasets show that DecFus outperforms existing centralized and decentralized federated learning methods under different settings.