When Is Rank-1 Enough? Geometry-Guided Initialization for Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning (PEFT) is a standard way to adapt multimodal large language models, yet extremely low-rank settings---especially rank-1 LoRA---are often unstable. We show that this instability is not solely due to limited capacity: in the rank-1 regime, optimization is highly sensitive to the update direction. Concretely, pretrained vision and text features form mismatched anisotropic regions, yielding a dominant "gap" direction that acts like a translation component and disproportionately steers early gradients under rank-1 constraints. Analyzing pretrained representations, we identify a modality-gap axis that dominates early gradient flow, while a random rank-1 initialization is unlikely to align with it, leading to weak gradients and training collapse. We propose Gap-Init, a geometry-aware initialization that aligns the rank-1 LoRA direction with an estimated modality-gap vector from a small calibration set, while keeping the initial LoRA update zero. Across multiple vision-language tasks and backbones, Gap-Init consistently stabilizes rank-1 training and can match or outperform strong rank-8 baselines. Our results suggest that at the extreme low-rank limit, initial alignment can matter as much as rank itself.

Large AI systems that understand both images and language are expensive to adapt to new tasks, because updating the whole model requires substantial computing resources. A common workaround is to update only a tiny part of the model, but when this trainable part becomes extremely small, training can become unreliable and sometimes fails completely. This paper studies why this failure happens. We find that the problem is not only that the trainable part is too small; it also matters whether it starts in a useful direction. Before adaptation, the model’s image and text information are not perfectly aligned, creating a major gap between the two. If the tiny update starts in a direction unrelated to this gap, the model receives weak training signals and may fail to learn. We introduce Gap-Init, a simple way to choose a better starting direction using a small set of examples. Across several image-language tasks and models, this makes extremely small updates much more stable, often matching or outperforming larger updates. Our results show that, for low-cost adaptation of multimodal AI systems, choosing the right starting direction can be as important as increasing the amount of training.