Controlled Collaboration Geometry for Personalized Federated Learning

In personalized federated learning (PFL), collaboration graphs specify model aggregation among clients. However, without constraints on the collaboration geometry, training can drift into two degenerate regimes: global consensus or spontaneous clustering. This paper provides a unified dynamical analysis: under the same budget of representative models, collaborative PFL is more expressive and achieves higher-order approximation accuracy than clustered PFL. An upper bound on disagreement further reveals two degeneration mechanisms—overly strong collaboration drives consensus (reducing to standard federated learning), while similarity-driven weight updates make the graph nearly reducible and induce self-clustering (collapsing to clustered PFL). Motivated by these findings, we propose pFedCCG. pFedCCG preserves the expressivity advantage via controlled collaboration geometry (CCG): it builds a static similarity-based collaboration template decoupled from training, optimizes a Markovian collaboration matrix with a prescribed stationary distribution via reversible parameterization and Euclidean projection, and schedules collaboration strength to avoid self-clustering. Experiments across diverse heterogeneity settings show consistent personalization gains and markedly reduced collapse and self-clustering. Code is available at https://github.com/YinHonb/pFedCCG.

Federated learning allows many devices or organizations to train machine learning models together without sharing their raw data. However, different users often have very different data, so a single shared model may not work well for everyone. Personalized federated learning tries to solve this by letting clients collaborate selectively with others that are more relevant to them. Our work studies an important but often overlooked problem in these collaboration systems. We show that existing methods can gradually break down in two extreme ways: either all clients become too similar and collapse into one global model, or the system splits into isolated groups that barely communicate with each other. In both cases, personalization is weakened. To address this, we propose a new method called pFedCCG that carefully controls how collaboration evolves during training. Instead of constantly changing collaboration based on current model similarity, our method builds a stable collaboration structure and gradually adjusts the strength of collaboration over time. Experiments on image classification, language tasks, recommendation systems, and large language model fine-tuning show that our approach consistently improves personalization while avoiding these collapse behaviors.