TIME: Tensor-Factorized Mixture-of-Experts with Intrinsic Routing for Lifelong Multimodal Knowledge Editing

Lifelong multimodal knowledge editing allows vision language models to continuously adapt to dynamic updates to avoid catastrophic forgetting. To mitigate interference between sequential updates, recent paradigms have shifted towards modular parameter isolation. However, this strategy faces a critical scalability bottleneck: accumulating dense parameter blocks can lead to excessive memory growth, and managing these independent modules often uses decoupled routing mechanisms, resulting in architectural redundancy. To address this issue, we propose TIME (Tensor-Factorized Intrinsic Mixture-of-Experts), a unified framework harmonizing parameter efficiency with structural self-routing. TIME parameterizes each knowledge edit as a compact CP-decomposed tensor, significantly reducing complexity compared to low-rank matrices. Furthermore, departing from auxiliary semantic retrievers, we introduce an intrinsic routing mechanism that utilizes the tensor's input factors to directly define the active subspace, effectively enabling expert parameters to serve simultaneously as the routing logic. Extensive experiments demonstrate that TIME achieves state-of-the-art performance on lifelong editing benchmarks while successfully reducing memory usage and inference latency.

Vision-language models can answer questions about images and text, but they often need to be updated when new facts or knowledge become available. A major challenge is that repeatedly updating such models may cause them to forget earlier updates or require more and more extra memory over time. Existing methods often solve this by giving each update its own separate module, but these modules can become expensive to store and slow to use as the number of updates grows. We propose TIME, a new method for helping vision-language models learn many updates over time in a more compact way. Instead of storing each update as a large block of parameters, TIME represents it using a compressed tensor structure, which greatly reduces memory cost. TIME also lets each update help decide when it should be used, so the model does not need a separate routing system to select among updates. Our experiments show that TIME can update models continuously while preserving previous knowledge, using less memory and achieving faster inference. This makes lifelong multimodal model editing more practical for real-world systems that must keep adapting after deployment.