ICML 2021

Timezone: »

Spotlight

Exploiting Shared Representations for Personalized Federated Learning

Liam Collins · Hamed Hassani · Aryan Mokhtari · Sanjay Shakkottai

Thu Jul 22 05:30 PM -- 05:35 PM (PDT) @

in Multi-task Learning 1 »

Deep neural networks have shown the ability to extract universal feature representations from data such as images and text that have been useful for a variety of learning tasks. However, the fruits of representation learning have yet to be fully-realized in federated settings. Although data in federated settings is often non-i.i.d. across clients, the success of centralized deep learning suggests that data often shares a global {\em feature representation}, while the statistical heterogeneity across clients or tasks is concentrated in the {\em labels}. Based on this intuition, we propose a novel federated learning framework and algorithm for learning a shared data representation across clients and unique local heads for each client. Our algorithm harnesses the distributed computational power across clients to perform many local-updates with respect to the low-dimensional local parameters for every update of the representation. We prove that this method obtains linear convergence to the ground-truth representation with near-optimal sample complexity in a linear setting, demonstrating that it can efficiently reduce the problem dimension for each client. Further, we provide extensive experimental results demonstrating the improvement of our method over alternative personalized federated learning approaches in heterogeneous settings.

Keywords: Algorithms · Multitask, Transfer, and Meta Learning

Author Information

Liam Collins (University of Texas at Austin)

Ph. D. student at UT Austin Electrical and Computer Engineering advised by Aryan Mokhtari and Sanjay Shakkottai. Princeton BSE '19.

Hamed Hassani (University of Pennsylvania)

I am an assistant professor in the Department of Electrical and Systems Engineering (as of July 2017). I hold a secondary appointment in the Department of Computer and Information Systems. I am also a faculty affiliate of the Warren Center for Network and Data Sciences. Before joining Penn, I was a research fellow at the Simons Institute, UC Berkeley (program: Foundations of Machine Learning). Prior to that, I was a post-doctoral scholar and lecturer in the Institute for Machine Learning at ETH Zürich. I received my Ph.D. degree in Computer and Communication Sciences from EPFL.

Aryan Mokhtari (UT Austin)

Sanjay Shakkottai (University of Texas at Austin)

Related Events (a corresponding poster, oral, or spotlight)