CLoVE: Personalized Federated Learning through Clustering of Loss Vector Embeddings

We propose CLoVE (Clustering of Loss Vector Embeddings), a novel algorithm for Clustered Federated Learning (CFL). In CFL, clients are naturally grouped into clusters based on their data distribution. However, identifying these clusters is challenging, as client assignments are unknown. CLoVE utilizes client embeddings derived from model losses on client data, and leverages the insight that clients in the same cluster share similar loss values, while those in different clusters exhibit distinct loss patterns. Based on these embeddings, CLoVE is able to iteratively identify and separate clients from different clusters and optimize cluster-specific models through federated aggregation. Key advantages of CLoVE over existing CFL algorithms are (1) its simplicity, (2) its applicability to both supervised and unsupervised settings, and (3) the fact that it eliminates the need for near-optimal model initialization, which makes it more robust and better suited for real-world applications. We establish theoretical convergence bounds, showing that CLoVE can recover clusters accurately with high probability in a single round and converges exponentially fast to optimal models in a linear setting. Our comprehensive experiments comparing with a variety of both CFL and generic Personalized Federated Learning (PFL) algorithms on different types of datasets and an extensive array of non-IID settings demonstrate that CLoVE achieves highly accurate cluster recovery in just a few rounds of training, along with state-of-the-art model accuracy, across a variety of both supervised and unsupervised PFL tasks.

Federated learning enables many users or organizations train machine-learning models together without moving their private data to one central place. When different users have very different kinds of data, one shared model may not work well for everyone, and personalization becomes necessary. A personalization approach is to group similar users and train a separate model for each group. However, these groups are not usually known in advance. We introduce CLoVE, a method for discovering these groups automatically and training one personalized model per group. Instead of looking directly at users' data, CLoVE asks each user to test several candidate models locally and report only how well each model performs on their data. These performance scores form a "loss vector" that acts like a fingerprint: users from the same hidden group tend to produce similar fingerprints, while users from different groups produce different ones. CLoVE clusters these fingerprints, trains one model per cluster, and repeats this process until the groups stabilize. Our theory shows why this approach can recover the right groups reliably in a standard setting, and our experiments across many datasets demonstrate that CLoVE accurately recovers the groups in only a few communication rounds while achieving strong model accuracy. Compared with prior methods, CLoVE stands out for its simplicity, the fact that it does not need carefully chosen starting models, and its applicability to both supervised and unsupervised settings.