Model-Dowser: Data-Free Importance Probing to Mitigate Catastrophic Forgetting in Multimodal Large Language Models

Fine-tuning Multimodal Large Language Models (MLLMs) on task-specific data is an effective way to improve performance on downstream applications. However, such adaptation often leads to a degradation in generalization on pretrained tasks, a phenomenon known as Catastrophic Forgetting. Existing methods that aim to mitigate this issue either become ineffective when fine-tuning deeper layers of the language decoder or scale poorly with increasing model size. To address these limitations, we propose Model-Dowser, a novel sparse fine-tuning approach for MLLMs. Model-Dowser measures a principled importance score for each model parameter with respect to pretrained generalization (prior to downstream adaptation) by jointly considering weight magnitudes, input activations, and output sensitivities. During fine-tuning, Model-Dowser selectively preserves high-importance parameters and updates the remaining. Comprehensive experiments on two representative MLLMs, LLaVA and NVILA, demonstrate that Model-Dowser effectively mitigates catastrophic forgetting and consistently outperforms prior methods, while remaining resource-efficient and scalable to multi-billion-parameter models.

Multimodal Language Models (MLLMs), such as image-understanding chatbots, are trained on vast amounts of data to develop broad, general capabilities. When researchers fine-tune these models for a specific task, the models often forget what they previously knew, a phenomenon called catastrophic forgetting. Existing solutions either fail when fine-tuning runs deep into the model or demand prohibitive amounts of GPU memory. We propose Model-Dowser, a method that identifies which model parameters matter most for preserving the model's general knowledge before any fine-tuning begins. To do this without requiring the original training data (which is often unavailable), we let the model generate its own synthetic inputs and measure how sensitive each parameter is to perturbation. Parameters that strongly affect the model's outputs are frozen during fine-tuning; only the less critical ones are updated to learn the new task. Model-Dowser consistently outperforms prior methods across diverse tasks and model architectures, while requiring no extra GPU memory beyond standard training. This makes it practical for deploying large multimodal AI systems that must be adapted to new tasks without sacrificing what they already know.