OPIC: Enhancing Language Model Merging via Optimizing In-Context Capability

Task-vector–based model merging enables low-cost, training-free multi-task learning for large language models, but suffers from severe performance degradation due to task conflict. Prior mitigation strategies largely rely on validation data for costly hyperparameter tuning, limiting both interpretability and practicality. We therefore propose OPIC, an evolutionary optimization–based model merging framework. Our preliminary experiments reveal that the degradation of In-Context Learning (ICL) capabilities is a primary driver of task conflict. Motivated by this insight, we formulate model merging as an optimization problem with ICL preservation as the objective. OPIC introduces a hierarchical refinement operators and optimizes it using self-generated data, effectively eliminating the reliance on external validation sets. Experimental results demonstrate that OPIC achieves an average performance retention of 80.73%, outperforming SOTA methods and improving by up to 11.1% over recent validation-free approaches. In addition, OPIC is compatible with existing merging pipelines, offering a new alternative solution for deploying without validation dependencies. Code is available at: https://anonymous.4open.science/r/OPIC-CFFE.

We often build specialized language models by fine-tuning them for different domains, but keeping many separate models can require large storage space and high inference cost. We wondered if these expert models could be merged into one model that keeps their different skills, much like combining several specialists into a single more versatile assistant. We found that a merged model often loses part of its ability to learn from examples given directly in the prompt. This ability, known as in-context learning, turns out to be closely connected to how well the merged model performs on its original tasks. Based on this observation, we designed OPIC, a method that treats model merging as a search problem: it automatically looks for better ways to combine expert models while preserving this prompt-based learning ability. Instead of relying on real validation data from each task, OPIC uses automatically generated data to guide the search. This makes model merging more practical when data is private, unavailable, or costly to collect, while helping one merged model retain more of the strengths of its specialized source models.