Federated Learning with Unlabeled Clients: Personalization Can Happen in Low Dimensions

Personalized federated learning has emerged as a popular approach to training on devices holding statistically heterogeneous data, known as clients. However, most existing approaches require a client to have labeled data for training or finetuning in order to obtain their own personalized model. In this paper we address this by proposing FLowDUP, a novel method that is able to generate a personalized model using only a forward pass with unlabeled data. The generated model parameters reside in a low-dimensional subspace, enabling efficient communication and computation. FLowDUP's learning objective is theoretically motivated by our new transductive multi-task PAC-Bayesian generalization bound, that provides performance guarantees for unlabeled clients. The objective is structured in such a way that it allows both clients with labeled data and clients with only unlabeled data to contribute to the training process. To supplement our theoretical results we carry out a thorough experimental evaluation of FLowDUP, demonstrating strong empirical performance on a range of datasets with differing sorts of statistically heterogeneous clients. Through numerous ablation studies, we test the efficacy of the individual components of the method.

Many machine-learning systems are trained using data spread across phones, hospital systems, or other devices, without sending private data to a central location. Due to differences in their data, a single shared model may not be effective for everyone, and each device may need a model tailored to its own data. However, to achieve this, most existing methods require each device to have labeled data, such as images with their correct categories, which can be expensive or unrealistic. We introduce FLowDUP, a method that can create a personalized model for a new device using only its unlabeled data. We use a model generator to represent the whole personalized model in a compact, low-dimensional form. This keeps computation and communication efficient while preserving the federated-learning principle that data stays on the device. Using a mathematical framework called PAC-Bayesian theory, we also explain why unlabeled devices can still help improve training. In experiments on several computer vision benchmarks, FLowDUP often produces more accurate personalized models than existing methods, especially when only a small fraction of devices have labeled data.