Federated Manifold Learning (FML): Tackling Domain Heterogeneity with Structural Knowledge Transfer

Federated Learning (FL) faces significant challenges due to domain heterogeneity, where data from different clients exhibit substantial statistical shifts that hinder the generalization of the global model. Although existing methods attempt to mitigate this by exchanging class prototypes, they fall short by representing an entire class's complex distribution with a single point. This oversimplification disregards the rich structural information within the data, especially across diverse domains. To address this limitation, we propose a paradigm shift from point-based representation to structure-based knowledge transfer. We introduce Federated Manifold Learning (FML), a novel framework that leverages perceptual manifolds—the intrinsic geometric structures of classes in the feature space—as rich knowledge carriers. In FML, clients transmit compressed manifolds, which are adaptively fused on the server using an attention-based Manifold Mutual Learning (MML) mechanism. This process enables domain-specific structures to learn from each other, creating a unified yet flexible global convergence target. Manifold-guided local training, enforced by a manifold approximation loss and a separation loss, further aligns local models with this global structure. Extensive experiments on the Digits and Office31 benchmarks demonstrate that FML substantially outperforms state-of-the-art methods, achieving accuracy improvements of up to 6.47%.

Federated learning allows multiple devices or organizations to train an AI model together without sharing their private data. However, this becomes difficult when each participant’s data looks very different. For example, one client may have clean handwritten digit images, while another may have colorful street-view digits with complex backgrounds. A model trained by simply combining information from these very different sources may fail to work well for all of them. This paper proposes Federated Manifold Learning, a new way to share richer knowledge between clients. Instead of describing each class with only one average example, our method shares a compact description of the overall shape and variation of that class in the model’s learned feature space. The server then combines these class structures from different clients so that each client can benefit from knowledge learned in other domains while still preserving useful local characteristics. Experiments on two widely used benchmarks show that this approach improves accuracy over existing federated learning methods. These results suggest that sharing structured knowledge, rather than overly simple summaries, can help federated models become more reliable when real-world data differs across clients.