FedEMoE: Improving Personalization on Heterogeneous Federated Learning via Elastic Mixture of Experts Architecture

Heterogeneous federated learning (HtFL) has emerged as a promising approach to address heterogeneity in local computational resources and data distribution. However, existing methods cause performance degradation of model personalization because personalized and generalized knowledge are either intertwined or dominated by one of them. To address this issue, we propose a novel Elastic Mixture of Experts (EMoE) architecture on HtFL, namely FedEMoE, decoupling personalization from generalization. Specially, FedEMoE employs a multi-scale feature extraction mechanism via personalized experts to enrich personalized knowledge. Furthermore, we design an elastic shared expert to break the transferred knowledge bottleneck across heterogeneous client models. The elastic shared expert can adaptively expand or shrink according to the status of each expert by the weight spectrum analysis, respectively. Extensive experiments across statistical and model heterogeneity settings demonstrate that FedEMoE significantly outperforms state-of-the art methods on the accuracy of each heterogeneous model over diverse datasets.

How to solve the heterogeneous Federated Learning in real world? In heterogeneous federated learning, different clients have varying data and computing power. Existing methods struggle because personalized knowledge (what works for a specific client) and generalized knowledge (what works across clients) become tangled, or one dominates the other, hurting model performance. We propose FedEMoE, a new elastic mixture-of-experts architecture that cleanly separates personalization from generalization. It uses personalized experts to extract multi-scale features for individual clients, plus an elastic shared expert that automatically expands or shrinks based on expert weight analysis, breaking the knowledge transfer bottleneck across heterogeneous models. Extensive experiments show FedEMoE significantly improves accuracy for each client’s model across diverse datasets, enabling more practical and robust federated learning in real-world settings where devices and data naturally differ. The elastic mixture-of-experts architecture will also use in other complex environment.