Timezone: »
We consider two federated learning algorithms for training partially personalized models, where the shared and personal parameters are updated either simultaneously or alternately on the devices. Both algorithms have been proposed in the literature, but their convergence properties are not fully understood, especially for the alternating variant. We provide convergence analyses of both algorithms in the general nonconvex setting with partial participation and delineate the regime where one dominates the other. Our experiments on real-world image, text, and speech datasets demonstrate that (a) partial personalization can obtain most of the benefits of full model personalization with a small fraction of personal parameters, and, (b) the alternating update algorithm outperforms the simultaneous update algorithm by a small but consistent margin.
Author Information
Krishna Pillutla (University of Washington)
PhD student at the University of Washington. Advisors: Zaid Harchaoui and Sham Kakade Interests: ML/Optimization, structured prediction, federated learning
Kshitiz Malik (Facebook)
Abdel-rahman Mohamed (Facebook AI Research (FAIR))
Michael Rabbat (Facebook)
Maziar Sanjabi (Meta AI)
Lin Xiao (Meta AI Research)
Related Events (a corresponding poster, oral, or spotlight)
-
2022 Poster: Federated Learning with Partial Model Personalization »
Thu. Jul 21st through Fri the 22nd Room Hall E #724
More from the Same Authors
-
2021 : Federated Learning with Buffered Asynchronous Aggregation »
John Nguyen · Kshitiz Malik · Ashkan Yousefpour -
2021 : Industrial Booth (Facebook) »
Kshitiz Malik · Ashkan Yousefpour -
2022 : Towards Better Understanding of Self-Supervised Representations »
Neha Mukund Kalibhat · Kanika Narang · Hamed Firooz · Maziar Sanjabi · Soheil Feizi -
2022 : BARACK: Partially Supervised Group Robustness With Guarantees »
Nimit Sohoni · Maziar Sanjabi · Nicolas Ballas · Aditya Grover · Shaoliang Nie · Hamed Firooz · Christopher Re -
2022 : Positive Unlabeled Contrastive Representation Learning »
Anish Acharya · Sujay Sanghavi · Li Jing · Bhargav Bhushanam · Michael Rabbat · Dhruv Choudhary · Inderjit Dhillon -
2023 : An Adaptive Method for Minimizing Non-negative Losses »
Antonio Orvieto · Lin Xiao -
2023 Poster: Analyzing Privacy Leakage in Machine Learning via Multiple Hypothesis Testing: A Lesson From Fano »
Chuan Guo · Alexandre Sablayrolles · Maziar Sanjabi -
2023 Poster: Identifying Interpretable Subspaces in Image Representations »
Neha Mukund Kalibhat · Shweta Bhardwaj · C. Bayan Bruss · Hamed Firooz · Maziar Sanjabi · Soheil Feizi -
2023 Poster: On the Convergence Rates of Policy Gradient Methods »
Lin Xiao -
2023 Poster: Text-To-Concept (and Back) via Cross-Model Alignment »
Mazda Moayeri · Keivan Rezaei · Maziar Sanjabi · Soheil Feizi -
2022 Poster: UNIREX: A Unified Learning Framework for Language Model Rationale Extraction »
Aaron Chan · Maziar Sanjabi · Lambert Mathias · Liang Tan · Shaoliang Nie · Xiaochang Peng · Xiang Ren · Hamed Firooz -
2022 Spotlight: UNIREX: A Unified Learning Framework for Language Model Rationale Extraction »
Aaron Chan · Maziar Sanjabi · Lambert Mathias · Liang Tan · Shaoliang Nie · Xiaochang Peng · Xiang Ren · Hamed Firooz -
2021 : Industrial Panel »
Nathalie Baracaldo · Shiqiang Wang · Peter Kairouz · Zheng Xu · Kshitiz Malik · Tao Zhang -
2020 : Technical Talks Session 2 »
Jinhyun So · Chong Liu · Honglin Yuan · Krishna Pillutla · Leighton P Barnes · Ashkan Yousefpour · Swanand Kadhe -
2020 Poster: On the Convergence of Nesterov's Accelerated Gradient Method in Stochastic Settings »
Mahmoud Assran · Michael Rabbat -
2019 Poster: TarMAC: Targeted Multi-Agent Communication »
Abhishek Das · Theophile Gervet · Joshua Romoff · Dhruv Batra · Devi Parikh · Michael Rabbat · Joelle Pineau -
2019 Oral: TarMAC: Targeted Multi-Agent Communication »
Abhishek Das · Theophile Gervet · Joshua Romoff · Dhruv Batra · Devi Parikh · Michael Rabbat · Joelle Pineau -
2019 Poster: Stochastic Gradient Push for Distributed Deep Learning »
Mahmoud Assran · Nicolas Loizou · Nicolas Ballas · Michael Rabbat -
2019 Oral: Stochastic Gradient Push for Distributed Deep Learning »
Mahmoud Assran · Nicolas Loizou · Nicolas Ballas · Michael Rabbat