LAVA: A Unified Framework for Finetuning Language and Vision Models

LoRA and its variants have attracted considerable attention because of their abilities to tune a negligible number of parameters while achieving comparable downstream performance. This success is largely attributed to the intrinsic low-rank structure of model parameter spaces, which allows LoRA to train two projection matrices to project weights into a low-dimensional subspace and then map them back. However, it does not consider how to explore this low-rank subspace sufficiently and may lose the expression ability accordingly. Moreover, when using LoRA to tune convolution layers, a flatten operation is required to convert tensors into matrices. We argue that this will degrade the model's performance. In this paper, we address this issue from a general parameter sub-space perspective: we present a unified **L**anguage **A**nd **V**ision **A**daption finetuning framework (called **LAVA**). Specifically, we verify the existence of low-rank subspaces in convolution layers empirically and propose to parameterize the increment of both convolution kernels and matrices as sum of learnable rank-1 components. To improve training stability, we analyze the optimization dynamics of LoRA and incorporate orthogonal regularization into our parameterization, for which we give theoretical proof that it will help reduce the variance of the gradient. We conduct various experiments on different downstreaming tasks to validate LAVA's superiority. For example, when tuning LLaMA2-7b for commonsense tasks, the performance of our LAVA is **+1.9\%** higher than that of LoRA. For metric depth estimation tasks, LAVA only tunes $\sim$1.5\% of Depth-Anything (335.3M), and achieves **+3.5\%** $\delta_1$ accuracy against that of LoRA and **+5.6\%** $\delta_1$ accuracy against that of SVDiff.

Large AI models are powerful, but adapting them to new tasks often requires substantial memory and computation. Existing efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA), reduce this cost by training only a small number of additional low-rank parameters. However, they may not fully use this limited training space, and they are less suitable for vision models that rely on convolution layers, where preserving the spatial structure of images is important. We introduce LAVA, a unified method for efficiently fine-tuning both language and vision models. Instead of treating all model components as the same flattened matrix, LAVA keeps the structure of convolution kernels and represents updates as a small set of learnable components. It also encourages these components to capture different information rather than becoming redundant, making training more stable. Across language understanding, commonsense reasoning, depth estimation, and image generation tasks, LAVA achieves stronger performance than LoRA while tuning only a small fraction of the model parameters. This makes adapting large AI models more practical and resource-efficient, especially when computing resources are limited.