FRISM: Fine-Grained Reasoning Injection via Subspace-Level Model Merging for Vision–Language Models

Efficiently enhancing the reasoning capabilities of Vision-Language Models (VLMs) by merging them with Large Reasoning Models (LRMs) has emerged as a promising direction. However, existing methods typically operate at a coarse-grained layer level, which often leads to a trade-off between injecting reasoning capabilities and preserving visual capabilities. To address this limitation, we propose FRISM (Fine-grained Reasoning Injection via Subspace-level model Merging), a fine-grained reasoning injection framework based on subspace-level model merging. Observing that different SVD subspaces contribute differently to reasoning and perception, FRISM decomposes LRM task vectors via Singular Value Decomposition (SVD) and adaptively tunes the scaling coefficients of each subspace through learning to realize fine-grained reasoning injection. Furthermore, we introduce a label-free self-distillation learning strategy with dual-objective optimization using common vision-language perception datasets. Extensive experiments demonstrate that FRISM effectively improves reasoning capabilities while largely preserving the model's visual capabilities by consistently achieving strong performance across diverse visual-language reasoning benchmarks.

Vision-language models can answer questions about images, but they often struggle when the question requires careful reasoning, such as solving a visual math problem or connecting several clues in a scene. At the same time, recent language reasoning models are much better at step-by-step reasoning, but they do not naturally understand images. A simple way to combine these strengths is to merge the two models, but existing merging methods often damage the model’s ability to understand visual information. We introduce FRISM, a method that transfers reasoning ability into a vision-language model in a fine-grained way. Instead of mixing whole layers of two models, FRISM focuses on the subspaces of delta parameters that carry different kinds of information and learns how much reasoning knowledge to add from each one. It does this without needing labeled visual reasoning data, using unlabeled vision-language examples to keep the original visual behavior stable. Across multiple model sizes and benchmarks, FRISM improves visual reasoning while largely preserving image understanding. This makes it a practical and efficient way to build stronger multimodal AI systems without expensive post-training.