Unlocking the Potential of Continual Model Merging: An ODE Perspective

Continual Model Merging (CMM) enables rapid customization of foundation models by sequentially incorporating task-adapted models without repeated retraining. However, existing merging rules usually update the deployed model through fixed algebraic or projection-based operations, providing limited control over how much previously accumulated knowledge should be retained relative to the incoming task model. This limitation leads to unstable retention and performance degradation in long task streams, and becomes more pronounced when tasks have heterogeneous utilities. We propose ODE-driven Merging (ODE-M), a controllable framework that formulates each continual merge as a trajectory in parameter space rather than a one-step endpoint update. Motivated by mode connectivity, ODE-M constructs a barrier-aware trajectory using a rectified time-dependent velocity field, where lightweight first-order feedback from a small calibration set suppresses loss-increasing motion while preserving progress toward the incoming model. The next merged model is then obtained by selecting an operating point along this trajectory through a utility-aware time schedule, providing an explicit mechanism for balancing retained historical knowledge and incoming task expertise. Extensive experiments on standard CMM benchmarks show that ODE-M consistently improves over strong continual merging baselines across CLIP ViT backbones, stream lengths, and heterogeneous task-utility settings.

Modern AI models are often adapted for many different tasks, which creates many specialized versions of the same original model. When new needs appear, it is costly to train a new model from scratch, so researchers would like to combine these specialized models into one model. A key challenge is that, when models are combined one after another, the combined model may lose useful abilities from earlier tasks or give too much attention to the newest task. This paper proposes a more careful way to combine such models over time. Instead of making a single abrupt update, our method treats each combination as a gradual transition from the current model toward the new task model. During this transition, it checks whether the update is likely to hurt performance and slows down harmful changes while still allowing the model to learn from the new task. It also controls how much influence the new task should have, which is important when some tasks matter more than others. Our experiments show that this approach produces stronger combined models than previous methods across different model sizes and task sequences. This can make it easier to maintain and update AI systems without repeatedly retraining them from scratch.