FedHPro: Federated Hyper-Prototype Learning via Gradient Matching

Federated Learning (FL) enables collaborative training of distributed clients while protecting privacy. To enhance generalization capability in FL, prototype-based FL is in the spotlight, since shared global prototypes offer semantic anchors for aligning client-specific local prototypes. However, existing methods update global prototypes at the prototype-level via averaging local prototypes or refining global anchors, which often leads to semantic drift across clients and subsequently yields a misaligned global signal. To alleviate this issue, we introduce hyper-prototypes, defined by a set of learnable global class-wise prototypes to preserve underlying semantic knowledge across clients. The hyper-prototypes are optimized via gradient matching to align with class-relevant characteristics distilled directly from clients' real samples, rather than prototype-level descriptors. We further propose FedHPro, a Federated Hyper-Prototype Learning framework, to leverage hyper-prototypes to promote inter-class separability via mutual-contrastive learning with client-specific margin, while encouraging intra-class uniformity through a consistency penalty. Comprehensive experiments under diverse heterogeneous scenarios confirm that 1) hyper-prototypes produce a more semantically consistent global signal, and 2) FedHPro achieves state-of-the-art performance on several benchmark datasets.

Federated learning lets many organizations or devices train a shared AI model without putting all their data in one place. However, these participants often have very different data: for example, one hospital, phone, or camera may see patterns that another never sees. Existing methods try to guide training by sharing prototypes, or representative summaries of each class, but simply averaging these summaries can produce misleading references when clients are very different. We propose FedHPro, a federated learning method that builds more reliable global references termed hyper-prototypes. Instead of only averaging client summaries, FedHPro learns these references by matching the learning signals produced by real data samples, without requiring access to the raw data itself. The method then uses these hyper-prototypes to make examples from the same class more consistent across clients, while keeping different classes more clearly separated. Across diverse federated learning settings, including differences in labels, data quantities, and visual domains, FedHPro improves performance over existing FL methods.