CoLA: Cross-Modal Low-rank Adaptation for Multimodal Downstream Tasks

Foundation models have revolutionized AI, but adapting them efficiently for multimodal tasks, particularly in dual-stream architectures composed of unimodal encoders, such as DINO and BERT, remains a significant challenge. Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA) enable lightweight adaptation, yet they operate in isolation within each modality, limiting their ability in capturing cross-modal interactions. In this paper, we take a step in bridging this gap with Cross-Modal Low-Rank Adaptation (CoLA), a novel PEFT framework that extends LoRA by introducing a dedicated inter-modal adaptation pathway alongside the standard intra-modal one. This dual-path design enables CoLA to adapt unimodal foundation models to multimodal tasks effectively, without interference between modality-specific and cross-modal learning. We evaluate CoLA across a range of vision-language (RefCOCO, RefCOCO+, RefCOCOg) and audio-visual (AVE, AVS) benchmarks, where it consistently outperforms LORA, achieving a relative gain of around 3\% and 2\%, respectively, while maintaining parameter efficiency. Notably, CoLA enables the first multi-task PEFT framework for visual grounding, bridging a key gap in efficient multimodal adaptation. Code is available at https://github.com/peterwisu/CoLA

A popular and widely used method for efficiently adapting large AI models to new tasks is LoRA. LoRA is typically used on a single model with one type of input. In multimodal tasks, two models are often combined to handle different types of input, such as an image with text, or a video with sound. LoRA treats each model on its own and does not take advantage of the fact that the two models are solving the same task together. We developed CoLA, which extends LoRA so the two models can exchange information as they process each input. The information they exchange is different each time, depending on the image, sound, or text given as input. CoLA consistently improves performance on tasks that combine vision with language or sound, such as locating objects in images from written descriptions or identifying what is making sound in a video, making powerful multimodal AI more efficient and accessible.