Shift-Dependent Asymmetry: Orthogonal Inverse Low-Rank Adaptation for Federated Medical Segmentation

Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of segmentation foundation models for medical imaging. However, most federated LoRA methods adopt a uniform aggregation rule, which breaks under the encoder–decoder asymmetry in medical segmentation: the encoder is dominated by appearance shifts, while the decoder is dominated by supervision variations. This mismatch entangles shared anatomy with site-specific biases and harms generalization. To address this, we propose Inverse Asymmetric Tuning (IAT). IAT aligns adaptation with heterogeneity sources by personalizing module-specific components in the encoder to absorb appearance shifts and in the decoder to accommodate site-dependent supervision, while retaining a shared pathway for transferable consensus. However, structural separation alone is insufficient under LoRA’s bilinear parameterization, where multiplicative coupling can still cause site-specific updates to leak into the shared direction. We therefore introduce a Subspace Orthogonality Regularizer that penalizes shared–local collinearity in the effective update space, mitigating leakage without extra communication. Experiments show consistent improvements over strong federated LoRA and parameter-efficient FL baselines.

Medical image analysis, such as identifying tumors or segmenting organs, often requires training AI models on data from multiple hospitals. However, privacy regulations prevent hospitals from sharing patient images directly. Federated learning addresses this by allowing hospitals to collaboratively train a shared AI model without exchanging raw data — each hospital trains locally and only shares model updates. A key challenge is that images from different hospitals look very different due to varying scanners, protocols, and annotation practices. Existing methods treat all parts of the AI model the same when combining updates from different hospitals, which we found to be problematic. Specifically, the part of the model that processes raw images is mainly affected by differences in how images are acquired, while the part that produces segmentation results is mainly affected by differences in how doctors annotate images. We propose a new training strategy that handles these two types of differences separately, assigning the right components to absorb each type of variation. We also introduce a mathematical constraint that prevents information specific to one hospital from leaking into the shared model. Experiments on multi-hospital datasets for histology and retinal image segmentation show that our approach achieves better accuracy than existing methods while remaining communication-efficient.