UniFLoW: Universal Multi-Modal Federated LoRA Fine-Tuning Framework with Analytical Aggregation

As Multimodal Large Language Models (MLLMs) continue to be trained, the availability of public data diminishes, limiting the possibility for further training and adaptation. However, private data remains an underutilized yet valuable resource. Federated Learning (FL) enables decentralized training on private data, yet extending it to MLLMs is challenging: heterogeneous client modalities induce architectural incompatibility, and full-parameter fine-tuning of billion-scale models incurs prohibitive communication costs. Parameter-efficient methods like LoRA alleviate these issues but introduce aggregation inconsistency, as averaged low-rank updates fail to recover the true global update faithfully. To address these issues, we propose **UniFLoW**(Universal multi-modal Federated LoRA fine-tuning framework With Analytical Aggregation), a unified federated framework that leverages pre-trained large language models and a multi-modal Encoder architecture, and our proposed Federated Aggregating Analytical Low-Rank Adaption$FedA^2$-$LoRA$). UniFLoW effectively utilizes fragmented client-side multi-modal data while $FedA^2$-$LoRA$ ensuring consistent aggregation. And modality-specific encoders and a II stage training strategy ensure effective integration of diverse modalities without overfitting. Experiments on text, image, and speech demonstrate that **UniFLoW** enables scalable, communication-efficient, and aggregation-consistent federated fine-tuning, with $FedA^2$-$LoRA$ achieving state-of-the-art performance compared to existing FedLoRA approaches. We envision UniFLoW as a promising solution to the growing scarcity of public data.

Large AI models are often adapted to new tasks using data from many users or organizations. However, in many real-world settings, this data cannot be collected in one place because of privacy, ownership, or communication constraints. This makes it important to train models collaboratively while keeping data local. Our paper studies how to make this collaborative training more reliable and efficient when only a small part of the model is updated. Existing methods can become unstable because different participants may update the model in conflicting directions, leading to slow progress or inconsistent results. We propose a new approach that better combines the updates from different participants so that training is more stable and the final model performs better. Across language and multimodal tasks, our method improves accuracy and produces more reliable answers compared with existing approaches. The results suggest that large AI models can be adapted more effectively in privacy-sensitive distributed environments, which may be useful for applications such as healthcare, education, and personalized assistants where data should remain under local control.