KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Controls

Large Multimodal Models encode extensive factual knowledge in their pre-trained weights. However, its knowledge remains static and limited, unable to keep pace with real-world developments, which hinders continuous knowledge acquisition. Effective knowledge injection thus becomes critical, involving two goals: knowledge adaptation (injecting new knowledge) and knowledge retention (preserving old knowledge). Existing methods often struggle to learn new knowledge and suffer from catastrophic forgetting. To address these challenges, we propose KORE, a synergistic method centered around KnOwledge-oRientEd controls. These controls are implemented through a two-stage optimization process: (1) KORE automatically converts individual knowledge items into structured and comprehensive knowledge to ensure that the model accurately learns new knowledge, enabling accurate adaptation. (2) KORE stores previous knowledge in the covariance matrix of LMM's linear layer activations and initializes the adapter by projecting the original weights into the matrix's null space, defining a fine-tuning direction that minimizes interference with previous knowledge, enabling powerful retention. Extensive experiments on various LMMs, including LLaVA-v1.5 (7B), LLaVA-v1.5 (13B), and Qwen2.5-VL (7B), show that KORE achieves superior new knowledge injection performance and effectively mitigates catastrophic forgetting.

Large Multimodal Models (LMMs) act like vast digital encyclopedias, but their knowledge is "frozen" in time once their training ends. This makes it hard for them to keep up with new real-world events. When we try to update them, they often struggle to learn the new information correctly or, even worse, they completely forget what they previously knew—a problem called "catastrophic forgetting." To fix this, we developed KORE, a method focused on "Knowledge-Oriented" controls. KORE works in two smart steps: first, it automatically organizes messy new data into structured lessons so the model can learn them accurately. Second, it uses a clever mathematical trick to identify the model's "old memories" and protects them, ensuring that new updates don't overwrite existing knowledge. Our tests show that KORE allows models to stay up-to-date while remaining reliable experts on what they learned before.