MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning

Real-world perception and decision making are inherently multimodal, integrating complementary signals across modalities. However, training multimodal models faces two main obstacles. First, collecting large-scale, well-aligned paired multimodal datasets is often impractical, making end-to-end multimodal training difficult. Second, existing multimodal representations frequently entangle information shared across modalities with modality-specific information, hindering interpretability and control. We introduce MultiLoReFT, an efficient and scalable low-rank representation fine-tuning framework for multimodal learning with pretrained unimodal models. MultiLoReFT extends low-rank adaptation to the multimodal setting and learns interpretable projection subspaces that decouple shared and modality-specific information. Across simulated and real-world benchmarks, it produces representations that support multimodal prediction while explicitly revealing how shared and modality-specific information is distributed across modalities.

MultiLoReFT is a framework for building multimodal AI models from pretrained models that already understand individual data types, such as text, audio, or images. Instead of training a large multimodal model from scratch, it efficiently fine-tunes these existing representations using a compact low-rank approach. Its main goal is interpretability: it separates information that is shared across modalities from information that is specific to each modality, making it clearer what each data source contributes and what patterns the model relies on. Across simulated and real-world tasks, MultiLoReFT supports accurate multimodal prediction while providing a more transparent view of how multimodal information is organized.